<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>Cloud Native Architecture – platform-engineering</title><link>https://deploy-preview-35--cncfarchitecture.netlify.app/tags/platform-engineering/</link><description>Recent content in platform-engineering on Cloud Native Architecture</description><generator>Hugo -- gohugo.io</generator><language>en</language><lastBuildDate>Thu, 11 Jun 2026 00:00:00 +0000</lastBuildDate><atom:link href="https://deploy-preview-35--cncfarchitecture.netlify.app/tags/platform-engineering/index.xml" rel="self" type="application/rss+xml"/><item><title>Architectures: GitOps-Native Multi-Cluster Kubernetes Platform for EU Data Sovereignty</title><link>https://deploy-preview-35--cncfarchitecture.netlify.app/architectures/obmondo/</link><pubDate>Thu, 11 Jun 2026 00:00:00 +0000</pubDate><guid>https://deploy-preview-35--cncfarchitecture.netlify.app/architectures/obmondo/</guid><description>
&lt;h2 id="relevant-projects">Relevant Projects&lt;/h2>
&lt;div class="row row-cols-1 row-cols-md-3 mb-4">
&lt;div class="col mb-4">
&lt;div class="card h-100">
&lt;div class="card-header">
Kubernetes
&lt;/div>
&lt;div class="card-body">
&lt;p class="card-text">
&lt;p>&lt;a href="https://www.cncf.io/projects/kubernetes/">&lt;img src="https://raw.githubusercontent.com/cncf/artwork/main/projects/kubernetes/icon/color/kubernetes-icon-color.svg" alt="kubernetes logo">&lt;/a>&lt;/p>
&lt;ul>
&lt;li>&lt;strong>Using since:&lt;/strong> 2019&lt;/li>
&lt;li>&lt;strong>Current version:&lt;/strong> 1.33.x&lt;/li>
&lt;/ul>
&lt;p>The runtime substrate for all workloads. Cluster API manages the lifecycle of every cluster across Hetzner bare metal, AWS, and Azure declaratively from Git. No cloud console required.&lt;/p>
&lt;/p>
&lt;/div>
&lt;/div>
&lt;/div>
&lt;div class="col mb-4">
&lt;div class="card h-100">
&lt;div class="card-header">
Argo CD
&lt;/div>
&lt;div class="card-body">
&lt;p class="card-text">
&lt;p>&lt;a href="https://www.cncf.io/projects/argo/">&lt;img src="https://github.com/cncf/artwork/raw/main/projects/argo/horizontal/color/argo-horizontal-color.svg" alt="argo logo">&lt;/a>&lt;/p>
&lt;ul>
&lt;li>&lt;strong>Using since:&lt;/strong> 2021&lt;/li>
&lt;li>&lt;strong>Current version:&lt;/strong> v3.x&lt;/li>
&lt;/ul>
&lt;p>The reconciliation engine for the entire fleet. A fix committed once to the shared KubeAid chart library propagates to every cluster on the next ArgoCD sync no manual per-cluster patching.&lt;/p>
&lt;/p>
&lt;/div>
&lt;/div>
&lt;/div>
&lt;div class="col mb-4">
&lt;div class="card h-100">
&lt;div class="card-header">
Prometheus
&lt;/div>
&lt;div class="card-body">
&lt;p class="card-text">
&lt;p>&lt;a href="https://www.cncf.io/projects/prometheus/">&lt;img src="https://raw.githubusercontent.com/cncf/artwork/main/projects/prometheus/icon/color/prometheus-icon-color.svg" alt="prometheus logo">&lt;/a>&lt;/p>
&lt;ul>
&lt;li>&lt;strong>Using since:&lt;/strong> 2020&lt;/li>
&lt;li>&lt;strong>Current version:&lt;/strong> v3.x&lt;/li>
&lt;/ul>
&lt;p>Generated per-cluster from a single Jsonnet vars file using kube-prometheus. Custom alerting rules for TLS expiry forecasting, backup SLIs, database replication lag, and Kubernetes ListWatch failures compose with upstream rule libraries without YAML merge conflicts.&lt;/p>
&lt;/p>
&lt;/div>
&lt;/div>
&lt;/div>
&lt;div class="col mb-4">
&lt;div class="card h-100">
&lt;div class="card-header">
Cilium
&lt;/div>
&lt;div class="card-body">
&lt;p class="card-text">
&lt;p>&lt;a href="https://www.cncf.io/projects/cilium/">&lt;img src="https://raw.githubusercontent.com/cncf/artwork/main/projects/cilium/icon/color/cilium_icon-color.svg" alt="cilium logo">&lt;/a>&lt;/p>
&lt;ul>
&lt;li>&lt;strong>Using since:&lt;/strong> 2022&lt;/li>
&lt;li>&lt;strong>Current version:&lt;/strong> v1.17.x&lt;/li>
&lt;/ul>
&lt;p>Network policies are embedded alongside every workload manifest in KubeAid. FQDN egress rules for every external API dependency ship with the chart. Operators do not write network policies — KubeAid writes them once and applies them to every cluster.&lt;/p>
&lt;/p>
&lt;/div>
&lt;/div>
&lt;/div>
&lt;div class="col mb-4">
&lt;div class="card h-100">
&lt;div class="card-header">
cert-manager
&lt;/div>
&lt;div class="card-body">
&lt;p class="card-text">
&lt;p>&lt;a href="https://www.cncf.io/projects/cert-manager/">&lt;img src="https://raw.githubusercontent.com/cncf/artwork/main/projects/cert-manager/icon/color/cert-manager-icon-color.svg" alt="cert-manager logo">&lt;/a>&lt;/p>
&lt;ul>
&lt;li>&lt;strong>Using since:&lt;/strong> 2021&lt;/li>
&lt;li>&lt;strong>Current version:&lt;/strong> v1.17.x&lt;/li>
&lt;/ul>
&lt;p>Defaults to DNS-01 ACME challenge validation with cloud-specific IAM scoping: IRSA on AWS, Workload Identity on Azure, API tokens on Hetzner. A CertificateNotReady alerting rule and external TLS expiry probe ship as KubeAid defaults added after a silent TLS expiry failure in production.&lt;/p>
&lt;/p>
&lt;/div>
&lt;/div>
&lt;/div>
&lt;div class="col mb-4">
&lt;div class="card h-100">
&lt;div class="card-header">
Velero
&lt;/div>
&lt;div class="card-body">
&lt;p class="card-text">
&lt;p>&lt;a href="https://www.cncf.io/projects/velero/">&lt;img src="https://raw.githubusercontent.com/cncf/artwork/main/projects/velero/icon/color/velero-icon-color.svg" alt="velero logo">&lt;/a>&lt;/p>
&lt;ul>
&lt;li>&lt;strong>Using since:&lt;/strong> 2021&lt;/li>
&lt;li>&lt;strong>Current version:&lt;/strong> v1.15.x&lt;/li>
&lt;/ul>
&lt;p>Backup schedules, IAM policies, and object storage (S3, GCS, Azure Blob) are provisioned at cluster bootstrap. Velero also exports the Sealed Secrets private key to external object storage at bootstrap — making DR proven at provision time, not during a drill.&lt;/p>
&lt;/p>
&lt;/div>
&lt;/div>
&lt;/div>
&lt;div class="col mb-4">
&lt;div class="card h-100">
&lt;div class="card-header">
Cluster API
&lt;/div>
&lt;div class="card-body">
&lt;p class="card-text">
&lt;p>&lt;a href="https://cluster-api.sigs.k8s.io/">&lt;img src="https://raw.githubusercontent.com/kubernetes-sigs/cluster-api/main/logos/icons/cluster.svg" alt="cluster api logo">&lt;/a>&lt;/p>
&lt;ul>
&lt;li>&lt;strong>Using since:&lt;/strong> 2022&lt;/li>
&lt;li>&lt;strong>Current version:&lt;/strong> v1.10.x&lt;/li>
&lt;/ul>
&lt;p>Manages cluster lifecycle declaratively across Hetzner bare metal, AWS, and Azure. MachineHealthChecks detect and replace unhealthy nodes automatically. Cluster topology is version-controlled in Git.&lt;/p>
&lt;/p>
&lt;/div>
&lt;/div>
&lt;/div>
&lt;div class="col mb-4">
&lt;div class="card h-100">
&lt;div class="card-header">
Helm
&lt;/div>
&lt;div class="card-body">
&lt;p class="card-text">
&lt;p>&lt;a href="https://www.cncf.io/projects/helm/">&lt;img src="https://raw.githubusercontent.com/cncf/artwork/main/projects/helm/icon/color/helm-icon-color.svg" alt="helm logo">&lt;/a>&lt;/p>
&lt;ul>
&lt;li>&lt;strong>Using since:&lt;/strong> 2019&lt;/li>
&lt;li>&lt;strong>Current version:&lt;/strong> v3.x&lt;/li>
&lt;/ul>
&lt;p>Over 100 curated Helm charts form the KubeAid chart library. Each chart ships with pre-wired Cilium network policies, Prometheus alerting rules, and ArgoCD ignoreDifferences configurations. Customers override only genuine differences.&lt;/p>
&lt;/p>
&lt;/div>
&lt;/div>
&lt;/div>
&lt;/div>
&lt;h2 id="tldr-synopsis">TLDR; Synopsis&lt;/h2>
&lt;p>This reference architecture describes how Obmondo operates production Kubernetes for dozens of customer clusters across four cloud providers and bare metal with a team of under 10 engineers using a two-repo GitOps pattern built entirely on CNCF projects.&lt;/p>
&lt;p>The core insight: every production failure fixed once in the shared platform repo (KubeAid) propagates to every cluster on the next ArgoCD sync. No cluster is ever patched manually. No cluster becomes a snowflake.&lt;/p>
&lt;p>This architecture targets:&lt;/p>
&lt;ul>
&lt;li>&lt;strong>Zero snowflake clusters&lt;/strong> — a fix committed once applies everywhere within hours via ArgoCD&lt;/li>
&lt;li>&lt;strong>Full EU data sovereignty&lt;/strong> — identical stack on Hetzner bare metal in Germany, on-premises datacentres, and EU-region cloud VMs, with no vendor control plane, no proprietary APIs, and full auditability&lt;/li>
&lt;li>&lt;strong>Disaster recovery by default&lt;/strong> — backup schedules, IAM, and Sealed Secrets key export are automated at cluster bootstrap; if a cluster can be created, it can be recovered&lt;/li>
&lt;li>&lt;strong>Sub-10-engineer fleet operations&lt;/strong> — platform-level abstractions that scale cluster count without scaling headcount&lt;/li>
&lt;/ul>
&lt;h2 id="organization">Organization&lt;/h2>
&lt;p>Obmondo (EnableIT ApS) is a Danish Managed Kubernetes provider. It builds and operates production Kubernetes platforms for customers in financial services, healthcare, and public sector organizations across Denmark and the EU. All infrastructure must satisfy GDPR, NIS2, and in many cases ISO 27001 — by architecture, not by policy.&lt;/p>
&lt;p>KubeAid, the open-source platform that powers Obmondo&amp;rsquo;s managed service, is available at &lt;a href="https://github.com/Obmondo/KubeAid">https://github.com/Obmondo/KubeAid&lt;/a> under the Apache 2.0 license.&lt;/p>
&lt;h2 id="teams">Teams&lt;/h2>
&lt;p>&lt;strong>Platform Engineering&lt;/strong> maintains KubeAid the shared chart library, Jsonnet monitoring templates, and ArgoCD application definitions that apply uniformly to every cluster. A fix here reaches every customer automatically.&lt;/p>
&lt;p>&lt;strong>SRE / Customer Operations&lt;/strong> handles day-2 operations: incident response, capacity planning, and customer-specific overrides in per-customer config repos. With KubeAid abstracting the platform layer, this team focuses exclusively on the ~10% that is genuinely different per customer.&lt;/p>
&lt;h2 id="architecture">Architecture&lt;/h2>
&lt;h3 id="goals">Goals&lt;/h3>
&lt;p>&lt;strong>Eliminate snowflake clusters.&lt;/strong> Every production failure becomes a KubeAid default. The silent TLS expiry became DNS-01 + external probe + alert. The duplicate Prometheus timestamp alert became a scrape-level relabeling rule. The RBAC gap became a ClusterRole fix. Each fix landed once; every cluster received it automatically.&lt;/p>
&lt;p>&lt;strong>EU data sovereignty without vendor lock-in.&lt;/strong> No proprietary control plane. No per-node SaaS fees. No APIs that cannot be audited. The entire stack runs on CNCF projects identically on Hetzner bare metal in Germany, on-premises in Danish datacentres, and in EU-region cloud VMs. Customers can move it, fork it, and audit every component.&lt;/p>
&lt;p>&lt;strong>Disaster recovery proven at provision time.&lt;/strong> Backup schedules, IAM policies, object storage, and Sealed Secrets private key export are automated at bootstrap. The DR gap was discovered simultaneously on every cluster during a planned drill and made impossible to miss on any future cluster.&lt;/p>
&lt;p>&lt;strong>Observability without per-cluster overhead.&lt;/strong> Monitoring configuration is generated from a single Jsonnet vars file. One engineer maintains alerting for the entire fleet. Custom rules compose with upstream libraries without merge conflicts.&lt;/p>
&lt;h3 id="architecture-overview">Architecture Overview&lt;/h3>
&lt;p>&lt;img src="images/obmondo-architecture.png" alt="Obmondo Platform Architecture">&lt;/p>
&lt;p>The architecture follows a strict two-repo pattern:
&lt;strong>Repo 1 — KubeAid (shared platform defaults)&lt;/strong>
Over 100 curated Helm charts with pre-wired integrations. ArgoCD ApplicationSets deploy these charts to every cluster. Monitoring is generated per-cluster from a single Jsonnet vars file using kube-prometheus. Cilium network policies ship alongside workload manifests. DR configuration is provisioned at bootstrap. Every production failure becomes a default here.&lt;/p>
&lt;p>&lt;strong>Repo 2 — Customer config (genuine differences only)&lt;/strong>
Each customer config repo holds only what is genuinely different: cloud provider, node sizes, alerting thresholds, compliance scope. If a value matches the KubeAid default, it does not exist in the config repo. ArgoCD reconciles both repos continuously across every cluster.&lt;/p>
&lt;p>&lt;strong>Per-cluster stack (all CNCF projects):&lt;/strong>&lt;/p>
&lt;table>
&lt;thead>
&lt;tr>
&lt;th>Layer&lt;/th>
&lt;th>Project&lt;/th>
&lt;th>Role&lt;/th>
&lt;/tr>
&lt;/thead>
&lt;tbody>
&lt;tr>
&lt;td>Lifecycle&lt;/td>
&lt;td>Cluster API&lt;/td>
&lt;td>Declarative cluster provisioning across all providers&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>Networking&lt;/td>
&lt;td>Cilium&lt;/td>
&lt;td>CNI + FQDN egress policies + Hubble metrics&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>TLS&lt;/td>
&lt;td>cert-manager&lt;/td>
&lt;td>DNS-01 ACME with cloud IAM scoping&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>GitOps&lt;/td>
&lt;td>ArgoCD&lt;/td>
&lt;td>Continuous reconciliation from both repos&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>Observability&lt;/td>
&lt;td>Prometheus + Alertmanager&lt;/td>
&lt;td>Generated from Jsonnet per cluster&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>Ingress&lt;/td>
&lt;td>Traefik&lt;/td>
&lt;td>Wildcard TLS, HTTP→HTTPS redirect&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>Backup / DR&lt;/td>
&lt;td>Velero&lt;/td>
&lt;td>Scheduled backups + Sealed Secrets key export&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>Secrets&lt;/td>
&lt;td>Sealed Secrets (Bitnami)&lt;/td>
&lt;td>Encrypted secrets in Git, key exported at bootstrap&lt;/td>
&lt;/tr>
&lt;/tbody>
&lt;/table>
&lt;h3 id="infrastructure-layer">Infrastructure Layer&lt;/h3>
&lt;p>Cluster API manages cluster lifecycle across all providers. The same declarative topology definition works on Hetzner bare metal (HCloud provider), AWS, and Azure. MachineHealthChecks detect unhealthy nodes and trigger automatic replacement. Node removal by CAPI is tracked and DaemonSet ghost pods are handled via alerting.&lt;/p>
&lt;p>On Hetzner bare metal, clusters run on dedicated physical servers with ZFS and Ceph storage. Network isolation is enforced by Cilium at the pod level. No hypervisor, no shared infrastructure with other tenants.&lt;/p>
&lt;h3 id="networking-layer">Networking Layer&lt;/h3>
&lt;p>Cilium is the CNI for every cluster. Every KubeAid Helm chart ships with a &lt;code>CiliumNetworkPolicy&lt;/code> alongside the workload manifest. FQDN egress rules for every external API dependency are pre-written and applied by ArgoCD. Operators do not write network policies they are a platform default.&lt;/p>
&lt;p>FQDN policy enforcement is critical for compliance: customers need to attest that workloads only communicate with approved external endpoints. Cilium&amp;rsquo;s Hubble metrics feed into Prometheus for network observability.&lt;/p>
&lt;h3 id="tls-and-certificate-management">TLS and Certificate Management&lt;/h3>
&lt;p>cert-manager defaults to DNS-01 ACME challenge validation. The switch from HTTP-01 was forced by a production failure: Traefik&amp;rsquo;s global HTTP→HTTPS redirect for HSTS compliance made HTTP-01 ACME validation permanently impossible every challenge request was redirected before reaching the solver pod. Certificates failed to renew silently for up to 90 days.&lt;/p>
&lt;p>DNS-01 with cloud-specific IAM scoping (IRSA on AWS, Workload Identity on Azure, API tokens on Hetzner) is now the only supported challenge type. An external TLS expiry probe and a &lt;code>CertificateNotReady&lt;/code> Alertmanager rule ship as KubeAid defaults.&lt;/p>
&lt;h3 id="observability-layer">Observability Layer&lt;/h3>
&lt;p>Prometheus and Alertmanager are generated per-cluster from a single Jsonnet vars file using kube-prometheus. Custom alerting rules compose with upstream rule libraries using Jsonnet merging semantics no YAML merge conflicts.&lt;/p>
&lt;p>Custom rules added after production failures:&lt;/p>
&lt;ul>
&lt;li>&lt;code>CertificateNotReady&lt;/code> — 30-day TLS expiry warning&lt;/li>
&lt;li>&lt;code>VeleroBackupMissed&lt;/code> — backup age SLI&lt;/li>
&lt;li>&lt;code>PrometheusKubernetesListWatchFailures&lt;/code> — RBAC-induced scrape gaps&lt;/li>
&lt;li>&lt;code>KubeDaemonSetMisScheduled&lt;/code> — CAPI node removal detection&lt;/li>
&lt;li>Database replication lag alerts per database type&lt;/li>
&lt;/ul>
&lt;p>One engineer maintains monitoring configuration for the entire fleet.&lt;/p>
&lt;h3 id="disaster-recovery">Disaster Recovery&lt;/h3>
&lt;p>Velero backup schedules, IAM policies, and object storage configuration are provisioned at cluster bootstrap. DR is not a follow-up task it is a bootstrap invariant.&lt;/p>
&lt;p>The Sealed Secrets private key gap was discovered during a planned DR drill: a key that exists only inside the cluster cannot survive cluster loss, and every sealed secret in Git becomes permanently unrecoverable ciphertext. The private key is now automatically exported to external object storage via Velero at bootstrap. This protection is mandatory it is impossible to create a new KubeAid cluster without it.&lt;/p>
&lt;p>Velero backup completion metrics are scraped by Prometheus. An alert fires if a backup has not succeeded within its scheduled window.&lt;/p>
&lt;h2 id="can-you-expand-on-why-you-are-using-those-projectsservices">Can you expand on why you are using those projects/services?&lt;/h2>
&lt;p>&lt;strong>ArgoCD over Flux:&lt;/strong> ApplicationSets allow a single ArgoCD instance to manage an arbitrary number of clusters from a centralized config. ArgoCD&amp;rsquo;s multi-source application support maps cleanly to the two-repo pattern — one source for KubeAid defaults, one for customer overrides. The ArgoCD UI provides fleet-wide visibility into sync status and drift.&lt;/p>
&lt;p>&lt;strong>Cilium over Calico/Flannel:&lt;/strong> FQDN-based egress policies are a hard compliance requirement. Cilium&amp;rsquo;s &lt;code>CiliumNetworkPolicy&lt;/code> with FQDN selectors is the only CNI that supports this at the policy layer without a proxy. Hubble metrics integrate directly with Prometheus. eBPF-based enforcement has lower overhead than iptables at scale.&lt;/p>
&lt;p>&lt;strong>cert-manager with DNS-01 exclusively:&lt;/strong> HTTP-01 is broken by Traefik&amp;rsquo;s global HTTP→HTTPS redirect, which cannot be disabled without breaking HSTS compliance. DNS-01 with cloud IAM scoping works identically on bare metal and cloud, and does not require inbound HTTP access.&lt;/p>
&lt;p>&lt;strong>Cluster API over cloud-specific tooling:&lt;/strong> Provider-agnostic cluster lifecycle in Git. The same declarative model covers Hetzner bare metal, AWS, and Azure. Rolling control plane replacements, MachineHealthChecks, and autoscaling are provider-agnostic concerns handled by CAPI — not bespoke automation per cloud.&lt;/p>
&lt;p>&lt;strong>Helm for chart packaging:&lt;/strong> The KubeAid chart library wraps upstream charts and embeds the platform layer (network policies, monitoring rules, ArgoCD ignoreDifferences) alongside the workload. Customers use standard Helm values overrides. No custom CRDs or operators required for the integration layer.&lt;/p>
&lt;h2 id="what-works-particularly-well">What works particularly well&lt;/h2>
&lt;p>&lt;strong>The two-repo pattern scales linearly.&lt;/strong> A platform fix committed to KubeAid propagates to every cluster within hours. A team of under 10 engineers operates dozens of production clusters across four cloud providers with no per-cluster patching.&lt;/p>
&lt;p>&lt;strong>Production failures become permanent fixes.&lt;/strong> Every incident is an opportunity to close the gap for every cluster simultaneously. The fleet never re-experiences the same failure in the same way.&lt;/p>
&lt;p>&lt;strong>Jsonnet for monitoring composition.&lt;/strong> kube-prometheus Jsonnet templates allow custom alerting rules to compose with upstream libraries without merge conflicts. One engineer maintains alerting for the entire fleet.&lt;/p>
&lt;p>&lt;strong>Cilium FQDN policies as a compliance primitive.&lt;/strong> Embedding FQDN egress rules alongside every workload manifest means network policy is not a separate compliance exercise it ships with the workload definition and is applied everywhere.&lt;/p>
&lt;p>&lt;strong>Velero key export as a bootstrap invariant.&lt;/strong> Making Sealed Secrets private key export mandatory at cluster creation means DR is never an afterthought. The constraint was added after finding the same gap on every cluster simultaneously during a drill.&lt;/p>
&lt;p>&lt;strong>Zero vendor control plane.&lt;/strong> The same CNCF stack runs on Hetzner bare metal in Germany, on-premises in Danish datacentres, and in EU-region cloud VMs. Customers can move it, fork it, and audit every component. GDPR, NIS2, and ISO 27001 requirements are satisfied by architecture.&lt;/p>
&lt;h2 id="what-needs-improvement">What needs improvement&lt;/h2>
&lt;p>&lt;strong>ArgoCD ignoreDifferences maintenance.&lt;/strong> Runtime drift Azure webhook injections, controller-managed fields, CRD caBundle rotation requires ongoing ignoreDifferences tuning in every affected chart. Each cloud provider introduces its own drift patterns. This operational overhead ideally belongs upstream in the charts themselves.&lt;/p>
&lt;p>&lt;strong>kube-prometheus regeneration across clusters.&lt;/strong> When a fix lands in a shared Jsonnet library, manifests must be regenerated for every affected cluster. This is currently a per-cluster manual step. A CI pipeline that detects library changes, regenerates all affected clusters, and opens PRs automatically would eliminate the gap.&lt;/p>
&lt;p>&lt;strong>Cluster API bare metal host pool management.&lt;/strong> Rolling control plane replacements require a spare host in the pool. When all bare metal hosts are occupied, a new control plane node cannot be provisioned and the rollout stalls. Better capacity planning automation is needed.&lt;/p>
&lt;p>&lt;strong>Sealed Secrets rotation.&lt;/strong> Sealed Secrets use asymmetric encryption keyed to a specific cluster key. Key rotation requires re-sealing every secret in the config repo. Tooling to automate re-sealing across the fleet is not yet in place.&lt;/p>
&lt;h2 id="what-sort-of-glue-had-to-be-developed">What sort of &amp;ldquo;glue&amp;rdquo; had to be developed?&lt;/h2>
&lt;p>&lt;strong>KubeAid chart library.&lt;/strong> The 100+ Helm chart wrappers that embed Cilium policies, Prometheus rules, and ArgoCD ignoreDifferences alongside upstream charts. These are the integration layer pre-built wiring that no operator writes from scratch for each cluster.&lt;/p>
&lt;p>&lt;strong>kube-prometheus Jsonnet library extensions.&lt;/strong> Custom libsonnet files that extend the upstream kube-prometheus library with Obmondo-specific alerting rules. These compose cleanly with upstream rules via Jsonnet merging semantics.&lt;/p>
&lt;p>&lt;strong>ArgoCD ApplicationSet templates.&lt;/strong> Parameterized ApplicationSet definitions that deploy the full KubeAid chart suite to any cluster whose config repo follows the two-repo pattern. Adding a new cluster is a config-repo operation, not a platform operation.&lt;/p>
&lt;p>&lt;strong>Sealed Secrets key export automation.&lt;/strong> A bootstrap step that exports the Sealed Secrets private key to external object storage via Velero immediately after cluster creation. This step is mandatory it cannot be skipped.&lt;/p>
&lt;p>&lt;strong>Prometheus alert composition templates.&lt;/strong> A set of reusable Jsonnet patterns for constructing SLI-based alerts (backup age, certificate validity window, database replication lag) that are consistent across the fleet without per-cluster duplication.&lt;/p>
&lt;h2 id="how-did-the-architecture-evolve">How did the Architecture Evolve&lt;/h2>
&lt;p>The architecture began as manually-managed clusters with per-cluster Helm deployments. The first pain point was drift: a fix applied to cluster A was not applied to clusters B–Z. The second was silent failures that were only discovered at impact.&lt;/p>
&lt;p>Each failure drove a platform default:&lt;/p>
&lt;table>
&lt;thead>
&lt;tr>
&lt;th>Failure&lt;/th>
&lt;th>Root Cause&lt;/th>
&lt;th>KubeAid Default Added&lt;/th>
&lt;/tr>
&lt;/thead>
&lt;tbody>
&lt;tr>
&lt;td>Silent TLS expiry (90 days)&lt;/td>
&lt;td>HTTP-01 blocked by Traefik redirect&lt;/td>
&lt;td>DNS-01 + CertificateNotReady alert + external probe&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>Duplicate Prometheus timestamps&lt;/td>
&lt;td>Two kubelet endpoints emitting same metric&lt;/td>
&lt;td>Scrape-level relabeling rule in kube-prometheus&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>Service discovery silent failure&lt;/td>
&lt;td>Missing RBAC on prometheus-k8s ServiceAccount&lt;/td>
&lt;td>ClusterRole fix in prometheus-k8s chart&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>DR gap across all clusters&lt;/td>
&lt;td>Sealed Secrets key not exported&lt;/td>
&lt;td>Mandatory Velero key export at bootstrap&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>snowflake clusters&lt;/td>
&lt;td>Per-cluster manual patching&lt;/td>
&lt;td>Two-repo GitOps with KubeAid defaults&lt;/td>
&lt;/tr>
&lt;/tbody>
&lt;/table>
&lt;p>Each fix landed in KubeAid once. Every cluster received it. No future cluster will hit any of these failures in the same way.&lt;/p>
&lt;h2 id="whats-next-for-your-architecture">What&amp;rsquo;s next for your architecture?&lt;/h2>
&lt;p>&lt;strong>Automated kube-prometheus regeneration.&lt;/strong> A CI pipeline that detects when a shared Jsonnet library changes, regenerates manifests for all affected clusters, and opens PRs automatically eliminating the manual per-cluster regeneration step.&lt;/p>
&lt;p>&lt;strong>Deeper OpenTelemetry integration.&lt;/strong> Distributed tracing across customer workloads, integrated with the existing Prometheus metrics pipeline for a unified observability experience.&lt;/p>
&lt;p>&lt;strong>Automated compliance evidence generation.&lt;/strong> Generating GDPR, NIS2, and ISO 27001 evidence artifacts directly from Prometheus metrics and Kubernetes audit logs compliance as a cluster output, not a manual exercise.&lt;/p>
&lt;p>&lt;strong>Broader FQDN egress coverage.&lt;/strong> Extending Cilium FQDN policy defaults to every KubeAid chart in the library, closing the remaining gap between workloads that have pre-wired policies and those that do not.&lt;/p>
&lt;p>&lt;strong>Sealed Secrets rotation tooling.&lt;/strong> Automation to re-seal every secret in a config repo after a cluster key rotation making key rotation a safe, routine operation rather than a risky manual process.&lt;/p>
&lt;h2 id="key-takeaways--lessons">Key Takeaways / Lessons&lt;/h2>
&lt;p>&lt;strong>Every manual fix is a fix that has not been applied everywhere.&lt;/strong> The two-repo GitOps pattern is not primarily about automation it is about making the gap visible. If a fix requires touching a per-cluster config repo, it is a signal that the fix belongs in KubeAid instead.&lt;/p>
&lt;p>&lt;strong>Silent failures are the expensive ones.&lt;/strong> TLS expiry, backup failures, and RBAC gaps were all silent no component logged an error until impact. The alerting investment (CertificateNotReady, VeleroBackupMissed, PrometheusKubernetesListWatchFailures) pays back in avoided incidents, not reduced noise.&lt;/p>
&lt;p>&lt;strong>Compliance by architecture, not by policy.&lt;/strong> GDPR, NIS2, and ISO 27001 requirements are satisfied by the architecture itself: no vendor control plane, full auditability, data residency by infrastructure choice, DR proven at bootstrap. Policy documents that reference the architecture are a consequence, not the mechanism.&lt;/p>
&lt;p>&lt;strong>The CNCF ecosystem composability is the product.&lt;/strong> ArgoCD reconciles Helm charts. Helm packages Cilium, Prometheus, cert-manager, Velero. Prometheus scrapes Cilium Hubble metrics, cert-manager certificate status, and Velero backup completion metrics. Each CNCF project does one thing well. KubeAid is the integration layer and it is open source because the problems it solves are not unique to Obmondo.&lt;/p>
&lt;p>&lt;strong>DR gaps are always discovered on every cluster simultaneously.&lt;/strong> If a DR gap exists, it exists everywhere because every cluster was provisioned from the same template. The corollary is equally powerful: close the gap once, and it closes everywhere.&lt;/p>
&lt;h2 id="discussion">Discussion&lt;/h2>
&lt;p>End user members may participate in the &lt;a href="https://github.com/cncf/tab/issues/139">discussion thread&lt;/a> for this architecture.&lt;/p></description></item><item><title>Architectures: ZEISS Vision Care - Order Fulfillment</title><link>https://deploy-preview-35--cncfarchitecture.netlify.app/architectures/zeiss/</link><pubDate>Tue, 17 Mar 2026 00:00:00 +0000</pubDate><guid>https://deploy-preview-35--cncfarchitecture.netlify.app/architectures/zeiss/</guid><description>
&lt;h2 id="relevant-cncf-projects">Relevant CNCF projects&lt;/h2>
&lt;div class="row row-cols-1 row-cols-md-3 mb-4">
&lt;div class="col mb-4">
&lt;div class="card h-100">
&lt;div class="card-header">
Kubernetes
&lt;/div>
&lt;div class="card-body">
&lt;p class="card-text">
&lt;p>&lt;a href="https://www.cncf.io/projects/kubernetes/">&lt;img src="https://raw.githubusercontent.com/cncf/artwork/main/projects/kubernetes/icon/color/kubernetes-icon-color.svg" alt="kubernetes logo">&lt;/a>&lt;/p>
&lt;ul>
&lt;li>&lt;strong>Using since:&lt;/strong> 2020&lt;/li>
&lt;li>&lt;strong>Current version:&lt;/strong> 1.32.3&lt;/li>
&lt;/ul>
&lt;p>Hosts &amp;gt; 200 microservices supporting order fulfillment processes on managed Kubernetes. Provides the core compute platform for containerized services.&lt;/p>
&lt;/p>
&lt;/div>
&lt;/div>
&lt;/div>
&lt;div class="col mb-4">
&lt;div class="card h-100">
&lt;div class="card-header">
Dapr
&lt;/div>
&lt;div class="card-body">
&lt;p class="card-text">
&lt;p>&lt;a href="https://www.cncf.io/projects/dapr/">&lt;img src="https://raw.githubusercontent.com/cncf/artwork/main/projects/dapr/stacked/color/dapr-stacked-color.svg" alt="dapr logo">&lt;/a>&lt;/p>
&lt;ul>
&lt;li>&lt;strong>Using since:&lt;/strong> 2020&lt;/li>
&lt;li>&lt;strong>Current version:&lt;/strong> 1.17.0&lt;/li>
&lt;/ul>
&lt;p>Provides common building blocks like service invocation, pub/sub, and state management across microservices; vendor-neutral abstractions enable portability across cloud providers.&lt;/p>
&lt;/p>
&lt;/div>
&lt;/div>
&lt;/div>
&lt;div class="col mb-4">
&lt;div class="card h-100">
&lt;div class="card-header">
KEDA
&lt;/div>
&lt;div class="card-body">
&lt;p class="card-text">
&lt;p>&lt;a href="https://www.cncf.io/projects/keda/">&lt;img src="https://raw.githubusercontent.com/cncf/artwork/main/projects/keda/icon/color/keda-icon-color.svg" alt="keda logo">&lt;/a>&lt;/p>
&lt;ul>
&lt;li>&lt;strong>Using since:&lt;/strong> 2021/2022&lt;/li>
&lt;li>&lt;strong>Current version:&lt;/strong> 2.19.0&lt;/li>
&lt;/ul>
&lt;p>Event-driven scaling. KEDA acts as an event-driven scaler; examples of triggers include message-broker queue depth and resource utilization.&lt;/p>
&lt;/p>
&lt;/div>
&lt;/div>
&lt;/div>
&lt;div class="col mb-4">
&lt;div class="card h-100">
&lt;div class="card-header">
OpenTelemetry
&lt;/div>
&lt;div class="card-body">
&lt;p class="card-text">
&lt;p>&lt;a href="https://www.cncf.io/projects/opentelemetry/">&lt;img src="https://raw.githubusercontent.com/cncf/artwork/main/projects/opentelemetry/icon/color/opentelemetry-icon-color.svg" alt="opentelemetry logo">&lt;/a>&lt;/p>
&lt;ul>
&lt;li>&lt;strong>Using since:&lt;/strong> 2020&lt;/li>
&lt;li>&lt;strong>Current version:&lt;/strong> Collector Contrib 0.145.0&lt;/li>
&lt;/ul>
&lt;p>Provides consistent instrumentation and exports telemetry to multiple targets.&lt;/p>
&lt;/p>
&lt;/div>
&lt;/div>
&lt;/div>
&lt;div class="col mb-4">
&lt;div class="card h-100">
&lt;div class="card-header">
Helm
&lt;/div>
&lt;div class="card-body">
&lt;p class="card-text">
&lt;p>&lt;a href="https://www.cncf.io/projects/helm/">&lt;img src="https://raw.githubusercontent.com/cncf/artwork/main/projects/helm/icon/color/helm-icon-color.svg" alt="helm logo">&lt;/a>&lt;/p>
&lt;ul>
&lt;li>&lt;strong>Using since:&lt;/strong> 2020&lt;/li>
&lt;/ul>
&lt;p>Package management and templating for Kubernetes deployments.&lt;/p>
&lt;/p>
&lt;/div>
&lt;/div>
&lt;/div>
&lt;/div>
&lt;h2 id="organization">Organization&lt;/h2>
&lt;p>ZEISS Vision Care produces spectacle lenses, instruments for refraction, and glasses adjustment equipment. Often, lenses are manufactured to an individual consumer&amp;rsquo;s prescription, effectively a batch size of one, which makes order fulfillment a multi-step coordination process.&lt;/p>
&lt;p>Our Business Enablement &amp;amp; IT team develops, designs, and operates the order fulfillment platform that underpins this process.&lt;/p>
&lt;h2 id="synopsis">Synopsis&lt;/h2>
&lt;p>We are reworking the order fulfillment process using a greenfield approach to modernize core systems. Our goal is a reliable, scalable platform that can evolve with business needs while remaining cost-efficient. The platform supports several key capabilities:&lt;/p>
&lt;ul>
&lt;li>Order routing: decide where the order should be produced.&lt;/li>
&lt;li>Order document generation: generate the necessary manufacturing and shipping documents for the order.&lt;/li>
&lt;li>Logistics routing: decide how to ship the order to the customer.&lt;/li>
&lt;li>Additional auxiliary domains as required.&lt;/li>
&lt;/ul>
&lt;h2 id="architecture-overview--goals">Architecture overview &amp;amp; Goals&lt;/h2>
&lt;h3 id="goals">Goals&lt;/h3>
&lt;p>The architecture is designed for future-proofing and scale. By leveraging event-driven scaling with KEDA and decoupled microservices via Dapr, the system is built to seamlessly absorb high-volume, global order loads as new processes are continuously migrated to the new platform.&lt;/p>
&lt;p>&lt;strong>Key Requirements:&lt;/strong>&lt;/p>
&lt;ul>
&lt;li>Modern cloud-based infrastructure&lt;/li>
&lt;li>High reliability and scalability&lt;/li>
&lt;li>Maintainable and extensible&lt;/li>
&lt;li>Cost-effective operations&lt;/li>
&lt;/ul>
&lt;h3 id="architecture-overview">Architecture Overview&lt;/h3>
&lt;p>&lt;img src="./images/solutionArchitecture.svg" alt="Architecture">&lt;/p>
&lt;h4 id="data--storage-strategy">Data &amp;amp; Storage Strategy&lt;/h4>
&lt;p>The platform uses Azure-managed data services as backing stores:&lt;/p>
&lt;ul>
&lt;li>&lt;strong>MSSQL&lt;/strong>: Transactional order data and relational schemas&lt;/li>
&lt;li>&lt;strong>Cosmos DB&lt;/strong>: Scaled state management and cross-region replication&lt;/li>
&lt;li>&lt;strong>Azure Blob Storage&lt;/strong>: Order documents and manufacturing files&lt;/li>
&lt;li>&lt;strong>Redis&lt;/strong>: Caching operations to improve latency and throughput&lt;/li>
&lt;/ul>
&lt;p>Dapr&amp;rsquo;s state abstraction layer is used for Cosmos DB, Azure Blob Storage, and Redis, decoupling microservices from those storage backends and enabling migrations without application-level changes. For Azure Blob Storage, we also use Dapr&amp;rsquo;s blob binding to interact with data, in addition to the state abstraction layer.&lt;/p>
&lt;h4 id="messaging--asynchronous-communication">Messaging &amp;amp; Asynchronous Communication&lt;/h4>
&lt;p>Azure Service Bus serves as the central message broker for asynchronous processes. Services publish and subscribe to topics via Dapr&amp;rsquo;s pub/sub building block, enabling loose coupling between applications. This event-driven approach provides resilience and allows each service to scale independently based on demand.&lt;/p>
&lt;h4 id="network--service-discovery">Network &amp;amp; Service Discovery&lt;/h4>
&lt;p>Services use Dapr&amp;rsquo;s service invocation building block instead of hardcoded endpoints, enabling resilience and the ability to replace or upgrade service implementations without changing callers. External traffic is routed to services via Ingress NGINX.&lt;/p>
&lt;h4 id="cicd--deployment">CI/CD &amp;amp; Deployment&lt;/h4>
&lt;p>Azure DevOps Pipelines automate building, testing, and deploying applications. We use a single, centralized Helm chart and store deployment configuration in Git as the single source of truth; the pipeline generates environment- and service-specific &lt;code>values.yaml&lt;/code> files and deploys each service as its own Helm release to Kubernetes.&lt;/p>
&lt;h2 id="can-you-expand-on-why-you-are-using-those-projectsservices">Can you expand on why you are using those projects/services?&lt;/h2>
&lt;p>We rely heavily on CNCF projects and open-source tooling to form the backbone of our platform:&lt;/p>
&lt;ul>
&lt;li>&lt;strong>Kubernetes (AKS)&lt;/strong> &lt;em>(Using since 2020)&lt;/em>: Hosts &amp;gt; 200 microservices supporting order fulfillment workflows. It provides the core compute platform for containerized services, offering a flexible model that supports multiple frameworks, programming languages, and dynamic scaling.&lt;/li>
&lt;li>&lt;strong>Dapr&lt;/strong> &lt;em>(Using since 2020)&lt;/em>: Provides common building blocks like service invocation, pub/sub, and state management across microservices. Dapr enables vendor-neutral capabilities, addressing service discovery and maintaining portability.&lt;/li>
&lt;li>&lt;strong>KEDA&lt;/strong> &lt;em>(Using since 2021/2022)&lt;/em>: Event-driven scaling based on Azure Service Bus queue depth and CPU/memory utilization. This allows the system to scale aggressively to match real-time demand while running economically during quieter periods.&lt;/li>
&lt;li>&lt;strong>Helm&lt;/strong>: Package management and templating. Charts are stored in Git, enabling reproducible deployments across environments since the project&amp;rsquo;s inception.&lt;/li>
&lt;li>&lt;strong>Ingress NGINX&lt;/strong> &lt;em>(currently in use, pending replacement – see Future Outlook)&lt;/em>: External traffic routing and load balancing.&lt;/li>
&lt;li>&lt;strong>OpenTelemetry Collector&lt;/strong> &lt;em>(Using since 2020)&lt;/em>: Provides consistent instrumentation and exports telemetry to multiple targets. It enables distributed tracing and metrics collection across microservices, ensuring observability and performance monitoring.&lt;/li>
&lt;/ul>
&lt;h2 id="what-has-worked-well">What has worked well?&lt;/h2>
&lt;p>Kubernetes, Helm, and KEDA have proven reliable and are widely used.&lt;/p>
&lt;p>&lt;strong>Scaling to Zero and Back:&lt;/strong>
We initially attempted to scale services to 0 replicas with KEDA to minimize costs, but this introduced operational challenges: delayed data flows during testing when pods needed to start up, and frequent scale churn during traffic pauses between messages. We learned that setting a minimum replica count for baseline load while scaling out for peaks was much more cost-effective and reliable for our specific workload patterns.&lt;/p>
&lt;p>&lt;strong>The Advantage of Abstraction:&lt;/strong>
A major architectural risk early on was building the entire system on Dapr (starting at pre-1.0 release 0.7.0). Early adoption carried organizational risk and we initially faced instability with actors under heavy load. Over time, those issues were fully resolved. The decision proved highly beneficial: Dapr’s maturity and vendor-neutral approach validated the initial decision, giving the platform extreme portability, flexibility, and saving custom boilerplate.&lt;/p>
&lt;p>&lt;strong>Platform Evolution:&lt;/strong>
Dapr is consistently evolving, allowing us to streamline our platform by replacing specific SDKs with built-in functionalities over time. For instance, we are transitioning from Azure App Configuration to Dapr&amp;rsquo;s Configuration Building Block. In hindsight, we would have standardized on CNCF native components sooner (e.g., migrating earlier to OpenTelemetry Collector and a CNCF ingress solution) to avoid replacing vendor-specific SDKs mid-project.&lt;/p>
&lt;p>&lt;strong>Data Ownership and Boundaries:&lt;/strong>
A critical lesson was the strict enforcement of data ownership. In a microservices architecture with over 200 services, we learned that clearly defining which service owns which data is non-negotiable. We found that any direct data access between services, bypassing their dedicated APIs, inevitably leads to tight coupling and maintenance challenges. Adhering to Domain-Driven Design (DDD) principles, where each service exposes its data only through a well-defined API, was essential for maintaining a scalable and evolvable system. This prevented a &amp;ldquo;distributed monolith&amp;rdquo; and ensured long-term architectural integrity.&lt;/p>
&lt;h2 id="what-sort-of-glue-have-you-had-to-develop">What sort of &amp;ldquo;glue&amp;rdquo; have you had to develop?&lt;/h2>
&lt;p>Since we had a greenfield start, we were not constrained by legacy endpoints or migration paths. However, managing over 200 microservices required strict boundaries and standardized communication patterns. We heavily utilized Domain-Driven Design (DDD) principles—specifically &lt;strong>Bounded Contexts&lt;/strong> to ensure each microservice owns its data and domain logic.&lt;/p>
&lt;p>To enforce these boundaries and standardize the &amp;ldquo;glue&amp;rdquo; between services, we developed:&lt;/p>
&lt;ul>
&lt;li>&lt;strong>Standardized Helm Templates&lt;/strong>: A unified set of Helm charts that abstract away the complexity of Kubernetes manifests and Dapr sidecar configuration. Developers provide application-specific KEDA &lt;code>ScaledObject&lt;/code> definitions and Dapr component definitions.&lt;/li>
&lt;li>&lt;strong>Common Libraries&lt;/strong>: While Dapr abstracts away many infrastructure concerns, we built thin, language-specific wrappers around the Dapr SDKs to enforce internal logging and error-handling standards.&lt;/li>
&lt;/ul>
&lt;h2 id="impact--results">Impact &amp;amp; Results&lt;/h2>
&lt;p>By adopting this cloud-native stack, we have built a highly scalable and future-ready order fulfillment system. A key driver of this success has been our extensive use of Dapr, which not only significantly reduced boilerplate code but also provided us with a high degree of vendor neutrality. This abstraction layer means we are not tightly coupled to specific cloud providers&amp;rsquo; SDKs, enabling an architecture that is portable, flexible, and robust enough for long-term growth.&lt;/p>
&lt;h2 id="whats-next-for-your-architecture">What&amp;rsquo;s next for your architecture?&lt;/h2>
&lt;p>We are constantly investigating how to run our services as efficiently and economically as possible. A near-term priority is replacing our existing Ingress NGINX setup. Because the Kubernetes project has &lt;a href="https://kubernetes.io/blog/2025/11/11/ingress-nginx-retirement/">officially announced the retirement of the ingress-nginx project&lt;/a>, we are actively evaluating alternatives—including NGINX Gateway Fabric, Envoy Gateway, and Traefik—while striving to preserve our routing behavior, TLS automation, and operational stability. Transitioning to the Gateway API is a key step in future-proofing our external traffic routing.&lt;/p>
&lt;h2 id="discussion">Discussion&lt;/h2>
&lt;p>End user members may participate in the &lt;a href="https://github.com/cncf/tab/discussions/135">discussion thread&lt;/a> for this architecture.&lt;/p></description></item></channel></rss>