Cloud Native Architecture – service-delivery

Architectures: A modern and sovereign Private Cloud «Kubernetes Service» for Swiss-based enterprises.

Wed, 04 Mar 2026 00:00:00 +0000

Relevant CNCF projects

Kubernetes

Using since: 2024
Current version: 1.32.8 (CNIP)
Current version: 1.31.x - 1.34.x (SKP)

Kubernetes enables high availability, scalability, and performance for infrastructure, offering a centralized and policy-driven platform to manage network and service data supporting Managed Kubernetes for our cloud customers.

KubeVirt

Using since: 2024
Current version: v1.5.0 (CNIP)

Cluster resources are constructed using KubeVirt for virtual machine abstraction of Control Plane and Worker instances.

Kube-OVN

Using since: 2024
Current version: v1.13.14 (CNIP)

Kube-OVN is utilized as network stack of the infrastructure cluster to enable intra-cluster/east-west network communication of user clusters. It enables a policy-driven security model as well as customer network isolation using VPCs.

MetalLB

Using since: 2024
Current version: v0.15.3

MetalLB is an integral component of the infrastructure deployment process, offering automated access to the framework that provisions individual user cluster resources on bare metal Kubernetes environments.

Container Storage Interface (CSI)

Using since: 2024
Current version: v25.06.3 (trident-csi)
Current version: v0.4.5 (kubevirt-csi)

Kubevirt-CSI is the standard storage interface for persistent volumes in user clusters. Trident-CSI manages NetApp storage requests and can also be used directly to integrate with Swisscom’s File Service Kubernetes, which provides iSCSI and NFS shared storage across all Availability Zones.

Kyverno

Using since: 2024
Current version: v1.13.4 (CNIP)

Kyverno serves as the default policy engine for infrastructure and user clusters, providing robust security constraints. In addition to Kyverno, also Chainsaw (a Kyverno sub-project) is used for automated, declarative e2e testing.

ArgoCD

Using since: 2024
Current version: v3.2.0 (CNIP)

ArgoCD allows us to deliver comprehensive infrastructure using a fully automated GitOps methodology.

Helm

Using since: 2024
Current version: v3.5.1 (CNIP)

Helm automates the creation, packaging, configuration, and deployment of Kubernetes applications by creating reusable charts.

CloudNativePG

Using since: 2024
Current version: v1.27.0 (CNIP)

CloudNativePG (CNPG) manages PostgreSQL databases in cloud-native environments. It handles the full lifecycle of highly available PostgreSQL clusters (primary/standby with native streaming replication), including declarative deployment, scaling, backups, self-healing, failover and monitoring.

Describe your organisation

Swisscom is the leading ICT company in Switzerland and offers mobile, Internet and TV, as well as comprehensive IT and digital services to private and business customers. Swisscom’s expertise in cloud native technologies is well-established, as evidenced by its status as a former Gold member and Management Board member of the Cloud Foundry Foundation, along with its certification for Cloud Foundry. Additionally, Swisscom demonstrates a strong commitment to the Open-Source community, having been a CNCF Silver Member for several years and serving as a Kubernetes Certified Service Provider (KCSP) partner. Our skilled employees have delivered numerous speeches and presentations at prestigious events such as KubeCon, Cloud Native Zürich, Swiss Cloud Native Day, KCD Suisse Romande, ContainerDays, among others.

Our next generation Private Cloud Container as a Service offering «Kubernetes Service» for the B2B market addresses customer’s need for scalable and highly available Kubernetes workload as a flexible and secure IT foundation. It is part of our Swiss-based Enterprise Service Cloud (ESC) market channel as a sovereign, Private Cloud Kubernetes offering for effortless provisioning and usage of our customer’s container workloads.

Describe your entity and/or team

The development and delivery of the new «Kubernetes Service» is done at within Swisscom’s IT-Clouds Value Stream and shared across two teams:

Pathfinders: responsible for the Cloud Native Infrastructure Platform (CNIP). CNIP handles the creation, delivery, and lifecycle management of the KubeVirt-based virtual machines (VMs). These VMs function as nodes for both the Control Plane and Workers. The VMs are ephemeral and can be re-created immediately in case of any failure. They are solely used to enable container-based workloads and do not act as standalone VMs.
Guardians: responsible for the Swisscom Kubernetes Platform (SKP), which runs on top of CNIP. It consists of the installation of Kubermatic Kubernetes Platform (KKP) for the customer tenant (environment) and the setup and support of the highly available Control Plane for any customer (user) cluster.

The layered approach allows Swisscom to manage technological aspects distinctly by segregating the cloud native infrastructure (managed by the Pathfinders team) from the Kubernetes platform (managed by the Guardians team). This strategy ensures considerable flexibility, permitting each layer to be combined or integrated with other technologies in the future.

Brief overview of your architecture and any potential goals you are trying to achieve with it?

Summary

Kubernetes Service is the successor to our current container offering, representing a significant shift towards a more cloud-native approach using advanced Open-Source technology. Currently bound to a vendor-specific implementation, Swisscom has opted to employ open-source tools for the development of cloud native products for customer use. This strategy aims to minimize dependencies and mitigate the risk of vendor lock-ins.

By adopting this architecture, Swisscom can uphold quality within the cloud native domain while maintaining a competitive pricing model due to reduced reliance on external licensing and subscription models. Furthermore, having the ability to develop, maintain, and operate all components internally enhances our decision-making processes and strengthens our roadmap capabilities.

Another important point is that our customers’ data will always remain within Switzerland and under Swiss law. Since we fully own the platform and do not rely on any external vendors, we can confidently guarantee true data sovereignty, hosted entirely on our premises without relying on vague marketing claims. Additionally, because Swisscom is not subject to the US Cloud Act or similar foreign regulations, no non-Swiss legislation can access the data.

Brief overview of architecture

A simplified high-level diagram describes Kubernetes Service, including multi-tenancy and security aspects:

As illustrated in the figure, two separate and independent user tenants, BLUE and RED, are established on shared resources (depicted in yellow), managed by the Kubernetes Infrastructure Cluster. The foundation for all virtual abstractions is the Consolidated Infrastructure (COI) in Swisscom’s data centers.

Each customer-specific environment comprises a management zone (MGMT Zone) and a workload zone. These zones address shared responsibilities, where Swisscom provides the Control Plane for each customer’s environment (illustrated in blue and orange in the next figure).

Customers have the flexibility to deploy workloads within the workload zone independently of the management resources as required. Furthermore, each customer is able to maintain multiple environments. This provides an alternative method for segregating workloads at the tenant level instead of the Kubernetes cluster level, thereby ensuring comprehensive isolation from the outset.

Goals and objectives

One of the primary objectives of the product refresh is to offer more desired features to customers. Compared to the current offering, enhancements include:

Upstream Kubernetes versions with faster updates
Node Autoscaling
Integrated Backup functionalities
Native Kubernetes Load Balancer
Modern customer self-service portal
Additional Kubernetes add-ons available via Application Catalog

Moreover, additional options are directly available to our customers:

Choose from different Container Network Interfaces (CNI)
Access persistent storage through kubevirt-csi

With KubeVirt providing abstraction, KVM is employed as the hypervisor on bare-metal servers. From the customer’s perspective (Customer X), the administrator of their user cluster manages all selections and abstractions shown in the figure below, enabling customers to make independent decisions, e.g. choosing a default CNI from the available options (Cilium, Canal, None).

In addition to technical improvements, we aimed to minimise reliance on external vendors and build a truly sovereign cloud solution that can compete with Public Cloud offerings, free from outside service dependencies. Our goal is for customers to run their Kubernetes workloads in our sovereign ESC Cloud, providing a comprehensive alternative to US hyperscalers - in terms of functionality and, most importantly, data privacy.

Can you expand on why you are using those projects/services?

Cloud-Native Implementation: Utilized CNCF projects and technologies to deploy a comprehensive stack consistent with a microservices-based architecture, resulting in enhanced scalability and operational agility.
Kubernetes for Orchestration: Adopted Kubernetes to manage containerized workloads, enabling automated deployment, scaling, and resilience on management as well as user cluster level.
Kube-OVN as network layer: With Kube-OVN as CNI on the infrastructure clusters and it’s VPC functionality, it allows customer environments to be fully segregated on a shared platform, providing maximum flexibility and strong security enforcement. It enables the teams to use familiar cloud-native development, operations, and debugging tools and skills.
KubeVirt for VM abstraction: A high-quality, Kubernetes-native virtual machine abstraction facilitates the deployment of container-based resources on a centralized cloud-native infrastructure platform, while maintaining flexibility for future use of VM resources.
Open-Source & Cost Efficiency: CNCF components deliver vendor-neutral, cost-effective solutions that are fundamental to container orchestration and observability. These open-source tools form the foundation of our sovereign cloud initiative, empowering internal teams to design customized architectures independently of third-party vendors. Utilizing CNCF technologies allows us to maintain flexibility, scalability, and comprehensive control over our cloud infrastructure, supporting our strategic objectives of autonomy and innovation.
Declarative & Configuration-Driven Approach: CNCF tools align with the low-code/no-code principle by enabling declarative configuration management.

What has worked well?

The implementation has eventually lead to the product launch of Kubernetes Service in August 2025, with some strong outcomes:

Layered Architecture for Enhanced Robustness: The integration of Cloud Native Infrastructure Platform (CNIP) and Swisscom Kubernetes Platform (SKP) forms the foundation of the new Kubernetes Service, enabling flexible handling as separate platform layers for streamlined future operations.
Vendor-Agnostic Production Platform: By eliminating proprietary technology, a resilient and adaptable foundation has been established to host managed Kubernetes clusters within Swisscom’s Private Cloud, ensuring a high degree of flexibility and scalability, as well as privacy.
Modern Cloud-Native Foundation: The implementation of Kubernetes to deliver managed Kubernetes clusters to end customers enables a unified cloud-native stack across all layers of responsibility, promoting consistency and efficiency.
Best Practice Design: Collaborating with Kubermatic, a modern Kubernetes platform was designed, incorporating the latest technologies such as KubeVirt and Kube-OVN, to ensure an enterprise-ready solution for end customers.
Operational Excellence: Equipping teams with essential cloud-native and Kubernetes expertise enhances the attractiveness of Swisscom’s tech stack to potential candidates and reinforces the company’s commitment to the Open-Source community.
Successful Internal Adoption: The Kubernetes Service was successfully launched as Swisscom’s internal Container platform, achieving significant traction with over 60% of workloads migrated within the first 9 months of operation.

What has not worked well?

While the architecture delivered significant improvements, several challenges emerged during implementation:

Enterprise-Readiness of Cloud Native Technologies: Despite successful scaling in test and internal production environments, many advanced cloud-native technologies faced difficulties when deployed in enterprise-grade settings for end customers (e.g., B2B market). This highlighted the need for further refinement and testing in real-world scenarios.

For KubeVirt, for instance, there are only limited real-world examples, best practices or reference designs available to draw upon for large-scale, production-grade business deployments. Switching fully to Kube-OVN as the main network layer also demands extra effort and is less straightforward than traditional network solutions with established production lifecycles.
Limited Support and Professional Services: The availability of professional support, particularly 24/7, for open-source and cloud-native technologies is limited. This poses challenges for enterprises seeking to adopt these technologies and provide services with guaranteed service levels (SLAs).

A possible solution is for more companies to provide professional support and make these services more transparent. Furthermore, the CNCF could introduce a “Certified Supporter” verification system to strengthen trust in firms that offer professional support.
Knowledge Gaps and Skills Requirements: Adopting new technologies demands specialized knowledge and expertise. In-house engineers required additional training and support to effectively maintain and troubleshoot products built on these technologies.
Customer Acceptance and Migration Challenges: Introducing a new platform based on modern technologies, without a proven track record in enterprise-grade deployments, required significant effort to educate customers, facilitate migration from legacy stacks, and promote the benefits of a sovereign cloud solution. This process demanded substantial resources and support to ensure a smooth transition.

What sort of “glue” have you had to develop to enable usage of your architecture ?

The reference architecture provides a strong foundation, making it practical and easy to use. The below elements were designed to simplify adoption, improve usability, and ensure seamless interaction across layers acting as the “glue”:

Unified Abstraction APIs: Developed APIs (Open Service Broker spec) that hide complexity and provide a consistent interface for orchestration and other consuming Operational Support Systems (OSS).
Advanced Routing Functionality: In order to integrate the customer environments into the Swisscom Core Network (MPLS), we developed and implemented our own concept of edge routers using BGP on FRRouting pods. This custom solution supports NAT, Fail-over (VRRP) as well as north-south firewalling (traffic from/to customer environments). These router pods are managed by an operator and configured with custom resource definitions.
Policy Integration Layer: Built operators to dynamically apply and manage Kyverno policies across different stages without requiring deep technical intervention.
Firewall Management: Implemented operators and API endpoints to allow customers to manage firewall rules on the SDN layer of the KubeVirt infrastructure, via Kube-OVN network policies.
Workflow Orchestration Logic: Developed and implemented the entire platform orchestration logic and automated pipelines from bottom-up.
Commandline tooling: Various commandline tools for human operators to manage and control the entire platform and all parts of it with ease.
Testing & Fine-tuning: With limited experience in large-scale bare-metal Kubernetes deployments, we had to do a lot of testing, validation and fine-tuning. We had to make sure, that the platform scales properly with more workloads being migrated every day.

Has your architecture evolved? What lessons have you learned from previous iterations?

Our architecture and product have undergone significant evolution through iterative development, driven by customer feedback and emerging requirements.

Iterative Development Approach: We began by establishing foundational layers and meeting the needs of our internal Swisscom customers. Subsequent iterations introduced advanced features for end customers, incorporating feedback from both internal and external stakeholders.
Continuous Improvement and Feedback Loop: Each iteration allowed us to gather valuable insights and add new functionalities, refining our product and enhancing customer satisfaction.
Steep Learning Curve and Expertise Development: As we ramped up the product, our teams faced a significant learning curve, developing essential expertise and professionalizing DevOps processes to ensure seamless operation.
Strategic Partnerships and Support: Our collaboration with Kubermatic enabled us to leverage professional support for key components, including KubeVirt and Kube-OVN, ultimately maturing our production platform and solidifying its readiness for enterprise-grade deployments.

Through this iterative process, we’ve gained valuable lessons and refined our architecture to better meet the needs of our customers, while developing the expertise and partnerships necessary to drive continued success.

Outcome

By embracing open-source and cloud native technologies, Swisscom successfully created a sovereign cloud solution, modernizing its container offering while reducing vendor lock-in and providing advanced features to customers. The new «Kubernetes Service» demonstrates the power of cloud native architectures in creating flexible, scalable, and cost-effective solutions for enterprise-grade services, all while ensuring true data sovereignty and regulatory compliance. This approach positions Swisscom as a leader in sovereign cloud solutions, offering Swiss (and European) customers a trusted alternative to global hyperscalers.

What’s next for your architecture? What are you looking to do next?

Building on the success of our proven reference architecture, which now supports both internal and external customer workloads in production, we’re focused on expanding and enhancing our offerings:

Hybrid Cloud Expansion and Multi-Cloud Flexibility: We’re working to enable seamless public cloud deployments, complementing our existing Swiss-based data centers and strengthening hybrid cloud use cases.
Edge Cloud Support: With cloud sovereignty in mind, we are developing a «Kubernetes Service On-Prem» extension that will deliver the Private Cloud product on a Cloud Edge Stack at customer premises, enabling an autonomous instance of our Kubernetes Service. This is currently in development with an interested customer.
GPU-Enabled Workloads and Emerging Technologies: Next, we’ll be integrating GPU support and exploring other emerging technologies to unlock new possibilities for compute-intensive applications.
Customer-Driven Features and Enhancements: We’re committed to delivering additional features and functionalities requested by our customers, further enriching our platform and services.
Simplified Onboarding and Resource Optimization: To improve efficiency and resource utilization, we’ll be introducing a shared cluster concept, allowing for more flexible and efficient use of our bare-metal infrastructure.
Exploring New Use Cases - VM Workloads: We’re also investigating the possibility of hosting classical VM workloads on our Cloud Native Infrastructure Platform (CNIP), expanding the platform’s use cases beyond container-based workloads and further increasing its versatility.

By pursuing these initiatives, we aim to continue delivering value to our customers, drive innovation, and grow our architecture and services to meet evolving needs.

Discussion

End user members may participate in the discussion thread for this architecture.

Architectures: Scaling Adobe’s Service Delivery Foundation with a Cell-based Architecture

Fri, 11 Oct 2024 00:00:00 +0000

Relevant CNCF projects

Kubernetes

Using since: 2019
Current version: 1.29.6

Kubernetes has been the foundation for our Internal Developer Platform and almost all of Adobe’s containerized workloads run on Kubernetes clusters.

Helm

Using since: 2019
Current version: v3.12.3

Helm is our package manager and helps us abstract out some of the complexity by including them as dependencies and only exposing the values.yaml file.

Argo

Using since: 2022

We are using all four projects under Argo:

Argo CD: v2.9.22
Argo Workflow: v3.4.6
Argo Events: v1.9.0
Argo Rollouts v1.6.0

Backstage

Using since: 2023
Current version: TBD

Backstage is the backbone for our unified internal developer portal.

Developer Platforms @ Adobe

Adobe has been working on building an Internal Developer Platform (IDP) for the past several years. We have a core infrastructure layer which offers a compute layer on top of cloud providers such as AWS and Azure. On top of the infrastructure layer, we provide an end-to-end developer experience which helps developers in different phases of SDLC.

Security, Compliance, Support and Cost Efficiency are cross-cutting concerns for both infrastructure and developer experience.

Developer Experience / Flex @ Adobe

Adobe offers a rich developer experience to its developers to enable them to write better software faster. This includes a seamless experience for all phases of their application lifecycle: Concept → Code → Cloud → Customer. Flex is the codename for Adobe’s GitOps-based Service Delivery foundation.

Concept → Code: Adobe offers standardized bootstrappable golden templates via the unified developer portal (based on Backstage), which provides a guided path that generates boilerplate code (with best practices built-in) for the app and infrastructure.
- The promise is that you should be able to run and test the app locally right after code generation.
- The generated infrastructure includes, among other things, the manifests for the service’s CI/CD pipeline.
- Toolchain - The developer portal is based on the industry-standard open-source tool Backstage.
Code → Cloud: Adobe offers customizable and flexible GitOps-based CI/CD pipelines (based on Argo projects) to developers.
- The developers start with a paved path but Flex allows users to customize their CI/CD pipeline if needed, including adding new steps or changing the pipeline DAG.
- The CI/CD pipeline does just-in-time provisioning during deployments i.e. it provisions necessary resources to make deployments successful. For example, DNS endpoints, K8s namespaces, Argo CD apps etc.
- Advanced deployment strategies like Canary, Blue-Green etc. are supported out of the box.
- Toolchain - We use industry-standard open-source Argo projects: Argo CD, Argo Workflows, Argo Events, and Argo Rollouts.
Cloud → Customer: Adobe offers a “Single pane of glass” developer experience (called Flexperience) in the developer portal for managing the application.
- Toolchain - The experience is provided as a Backstage plug-in in the developer portal.
Control Plane – The control plane for the service delivery workflow is based on AWS EKS clusters.

Flex has had rapid organic growth in the last 2 years since its launch. Here are some numbers that represent the scale that we are handling (as of Oct 2024):

> 360 remote K8s clusters
> 22K Argo CD apps
> 30K deployments per month
> 1K services in production

And these numbers are increasing by the day.

In this article, we will cover Flex’s initial architecture, its challenges, and the need for a re-architecture. Then, we will deep-dive into our new Flex-in-a-box architecture, including the architecture requirements and an introduction to cell-based architecture. We’ll conclude by discussing the benefits, challenges, use cases, and future ideas.

Initial Architecture

Here is a simplified high-level diagram describing how Flex acts as the foundation between the source (Github), destinations, and the developer surfaces.

As you can see in the diagram, the CI/CD foundation acts as the glue between the source and the destinations. The developer surfaces, powered by the foundation, offer paved roads and seamless workflows for various use cases.

Here is the initial architecture that we started with.

We have a Hub-and-Spoke architecture with the Hub in the middle (aka Flex Hub k8s cluster) connected to the various remote K8s clusters, which act as the spokes. As you can see, we have Git as the source of truth on the left, and K8s clusters as the remote clusters (destinations) on the right.

We use Argo Workflows for creating the CI/CD pipeline, Argo CD for providing GitOps capabilities, and Argo Events for catching events and triggering workflows. The CI/CD pipeline runs on the Hub cluster in a per-service namespace. Argo Rollouts is installed on remote K8s clusters and provides advanced deployment capabilities with Argo CD. The service runtime runs on namespaces provisioner on these remote K8s clusters.

Provisioner is a homegrown component that does the just-in-time provisioning during deployments. A couple of home-grown Observability components send events and information to our homegrown Flex backend, where the data is aggregated and exposed via a set of Flex APIs. The Flex developer experience (aka Flexperience) in the Developer Portal / Backstage uses the Flex APIs provided by the Flex backend to populate the service’s info.

Each service has two repos associated with it: an app repo and a deploy repo. App repo has the code for business logic, and the deploy repo has the configs and manifests for CI/CD, K8s resources etc.

Challenges with Initial Architecture

While the initial architecture provided a starting point, it had its challenges. We realized these challenges after 6 months when we crossed approximately 2.5K Argo CD applications.

No tuning - We used the Argo components with no/minimal tuning, and realized that there were several knobs and controls available on the Argo components that could be leveraged to improve the stability, performance, and scalability of the components.

Heavy load on K8s control plane - Flex Hub cluster is based on AWS EKS, and we just had one Hub cluster. We realized that Argo CD and Argo Workflows interact heavily with the EKS control plane, which includes the K8s API Server and etcd. The CI/CD experience is severely impacted when the K8s control plane is under heavy load.

Shared control plane - Both Argo CD and Argo Workflows were running on the same EKS cluster, which multiplied the overhead and load on the EKS control plane. This impact was observed in terms of degraded latency of the K8s API server, high etcd usage/churn etc., that caused a severe impact on the performance of anything interacting with the K8s control plane, including the Argo components. As a result, the user experience was getting impacted and client deployments were either taking a long time or failing.

No automation to create Flex Hub cluster - Flex Hub cluster was brought up manually, over many quarters, by different individuals. We did not have a concrete list of steps and automation to bring up a Flex Hub cluster, thereby putting us at risk if the Hub cluster goes down. It was a huge risk for the platform team and client teams because it would have taken us a long time to create a new one, and client deployments would have been stuck during that duration.

As a result of these challenges, we experienced several outages for the developer experience CI/CD workflows, including slow builds, hung workflows, and deployments taking too long, among other things.

Potential Solutions

Given the problems we faced, we realized quickly that one Flex Hub cluster would not be enough. We invested heavily in vertically scaling Flex Hub cluster by tuning and optimizing the Argo components, EKS control plane, and homegrown tooling. This helped us get out of the woods and support >10K Argo CD apps. While we will continue to scale Flex Hub cluster vertically to support more services/Argo CD apps, we knew that the runway will be limited.

We were projecting 10X growth in the next two years, and one Flex Hub cluster could not support all that scale alone. We needed more than one Flex Hub cluster.

Scale Up

There were a lot of configuration changes and optimizations we did to scale Flex Hub cluster vertically. These included:

Scale up replica count for Argo CD application controller and repo server
Ask Argo CD to exclude resources that we should not manage
Tune Argo CD self-heal and reconciliation timeouts
Adjust Argo CD repo server parallelism
Enable gzip compression for server responses
Archive old Argo Workflows by setting a short TTL
Switch from K8s APIs to Argo CD APIs to leverage informer cache

This list is not exhaustive and there were many more improvements which helped us tremendously. We have talked about some of these improvements in previous talks:

Scale Out

While the scale-up improvements were helping, we decided to re-architect. Though we had a couple of options, we decided to scale horizontally by having more than one Flex Hub cluster because it satisfied many of the architecture requirements (listed below) and gave us the biggest bang for the buck.

This was not going to be easy, because we had hundreds of production services running and we did not want to impact any of them. The rest of the article goes into the details of how we approached the re-architecture.

Flexbox concept

As a first step, we came up with the concept of a Flexbox. A Flexbox is a collection of Flex components that work together to provide the GitOps foundation for our CI/CD solution. We called the Flex Hub K8s cluster which formed the basis of our initial architecture, Flexbox 1.

Keeping future extensibility in mind, we also decoupled the concept of Flexbox from clusters. A Flexbox can be extended to support many physical and virtual Hub clusters inside it. Because the initial architecture had just one physical Flex Hub cluster, Flexbox 1 also had one Hub cluster.

New Architecture Requirements

We came up with the following key requirements for the new architecture.

Ability to predictably add scale (horizontally and vertically) as needed, for short-term and future needs.
Ability to relocate services out of Flexbox 1 and relieve pressure.
The services should not decide which Flexbox they are on.
No impact of rearchitecture on any existing services (runtime or CI/CD) on Flexbox 1.
No impact on service runtime during box-2-box relocation.
A service (app and deploy repos) should be associated with one Flexbox at a time.
Minimal impact/downtime during relocation to the service’s CI/CD pipeline, if at all.
During relocation, the service owners should be aware of the relocation and the downtime.
Ability to support more than one physical or virtual clusters inside a Flexbox.
Ability to support non-container workflows as well e.g. Serverless, Cloud Infrastructure etc.

What is Cell-Based architecture

Cell-based architecture is an architecture pattern that involves designing systems with multiple cells capable of executing the designated tasks independently. The basic idea is to have the ability to scale out by adding identical cells that handle the same kind of workload and isolate fault boundaries to limit the impact in case of failures.

Each cell is independent, does not share any state with other cells, and handles a subset of the overall workload requests. This reduces the potential impact of a failure, such as a bad software update, on an individual cell and the requests that it’s processing.

Here’s a diagram that depicts the cell-based architecture at a high level:

More details here: Diagram Source: https://newsletter.systemdesign.one/p/cell-based-architecture
Reference: https://docs.aws.amazon.com/wellarchitected/latest/reducing-scope-of-impact-with-cell-based-architecture/what-is-a-cell-based-architecture.html

Flex-in-a-Box (FiaB) architecture

With the requirements above, we developed a Flex-in-a-box (FiaB) architecture. This architecture allows us to encapsulate Flex components in boxes and easily replicate them to add scale on demand. We started by adding a new Flexbox 2. Before FiaB, all services were associated with Flexbox 1. This means that the CI/CD pipelines for these services run on Flexbox1. With FiaB and newer Flexboxes, now services can be onboarded to Flexbox 2 also, and any of the future Flexboxes.

To keep the impact of architectural changes to a minimum, we decided to have just one physical Hub cluster in Flexbox 2 too, and make it very similar to the Flex Hub cluster in Flexbox 1.

Our FiaB architecture is based on three Rs:

Recreation – Ability to (re)create and upgrade more Flexboxes with repeatability and predictability, with a GitOps-based CI/CD pipeline.
Redirection – Ability for Flex admins to decide which service maps to which Flexbox via declarative rules, and have services routed to the correct boxes based on those rules.
Relocation – Ability to relocate a service (with automation) from one Flexbox to another with zero downtime for the service runtime, zero work for the service team, and minimal downtime (< 15 mins) for the CI/CD pipeline.

Here’s a diagram that captures these three concepts. More details on these 3 Rs below.

Recreation

We created a GitOps-based pipeline for Flexbox lifecycle management. This means that there is a Flexbox config in Git which defines what a Flexbox looks like. Here is a diagram showing the workflow in action:

As can be seen from the diagram, any changes to the Flexbox config triggers a Flexbox automation Argo workflow. The workflow has a series of steps that automate the setup of a Flexbox on top of an EKS cluster.

The workflow has two key objectives:

Creates the Helm charts for the various Flexbox components - This step understands the Flexbox configs and creates the Helm charts for Flexbox components, including Argo CD, Argo Workflows, Argo Events, Provisioner and Monitoring.
- The workflow step then commits these Helm charts to Git, allowing Argo CD apps to sync these changes to the destination Flexbox.
Provisioning resources for Flexbox components, including creating Argo CD apps for each of the Flex components (Argo CD, Argo Workflows, Argo Events etc.) that need to be installed in the Flexbox. It’s an Argo of Argos architecture, where an Argo CD 0 (level zero) manages the Argo CD apps (level 1) for the various Flex components to be deployed on the destination Flexbox.

Redirection

Once we have a new Flexbox, we need the ability to map services to Flexboxes and redirect the Github events to the correct Flexbox. This function is performed by a new component called Redirector, which sits between Github and the Flexboxes (as can be seen in the new architecture diagram above).

Loosely speaking, Redirector acts as a poor man’s load balancer for Flexboxes. When the first event arrives for any service, Redirector evaluates the rules defined by Flex admins and maps the service to a Flexbox based on the rules. These rules are specified declaratively in a Github repo for Redirector by Flex admins. Redirector stores the box<>service mapping in a DB for future reference. For any subsequent GitHub events, it just picks up the mapping from the DB and routes the events to the appropriate Flexbox.

Relocation

While Flexbox 2 and other components in FiaB architecture were being built, there was constant organic adoption of Flex, Flexbox 1 was getting filled up and was under increasing pressure.

To relieve the pressure on Flexbox 1 and future use cases too, we needed a way to cleanly relocate a service from a source Flexbox to a destination Flexbox, without disrupting the service runtime. We created a new component called Relocator which does just that. Here are the four broad phases in the Relocator:

Stop the service’s CI/CD pipeline on source Flexbox - This would include pausing the Argo CD sync and stopping Redirector from sending the events for this service to the source Flexbox. This step also flags the service status (in Redirector) as “relocating” which helps us inform the developers that their service’s deployment pipeline is “under maintenance”.
Recreate the service’s CI/CD pipeline on the destination Flexbox - This includes recreating/copying the resources in the source Flexbox on the destination Flexbox.
Validate and start the service’s CI/CD on destination Flexbox - This includes starting the Argo CD sync and allowing Redirector to send the events for this service to the destination Flexbox.
Delete Flex pipeline on source Flexbox - Once everything is working fine, delete the resources on the source Flexbox to clean things up. This step also removes the “relocating” service status in Redirector.

Flex-in-a-box (FiaB) vs Cell-based architecture

When we started on FiaB, we did not use cell-based architecture as a reference. We realized later when we had conceived the new architecture that it is conceptually very similar to cell-based architecture.

In terms of comparison, a Cell in a cell-based architecture is loosely equivalent to a Flexbox in Flex-in-a-Box (FiaB).

Benefits / What worked well

The new Flex-in-a-box architecture offers various benefits:

“Stamp” setup - Ability to predictably and reliably create new Flexboxes.
Scale horizontally - Industry-proven way to scale horizontally by adding scale on demand.
Reduce existing load - Ability to relieve pressure on a Flexbox by allowing us to relocate services across Flexboxes, without any downtime.
Minimal relocation impact - Minimal impact/downtime to the service’s CI/CD pipeline for the duration of the relocation.
Abstracted Flexbox mapping - Service teams need not worry about which Flexbox they are on. Flex team decides the service<>Flexbox mapping.
Dedicated Flexboxes - Ability to provide dedicated Flexboxes to teams if need be.
Enhanced testing ability - Allow Flex team to test features/rollouts/infrastructure better using multiple boxes.
Configuration flexibility - Different Flexboxes may have different versions of Flex components installed if need be. In fact, different Flexboxes may have different components altogether. For example, one may host GitHub Action runners inside a Flexbox, rather than Argo Workflows. One Flexbox may have two clusters in it. We need to be wary of snowflakes though.
Better audit - Ability to track, monitor and audit GitHub events coming via Redirector.
Efficiency - Ability to maintain Flexboxes at optimum capacity rather than stretching one Flexbox to the limit.
Limit blast radius - If one of the boxes goes down, only services mapped to that Flexbox are affected, and can be relocated.
Better disaster recovery - Ability to recover quickly from disasters because of relocation.

Performance Tests / Benchmarking

We did several benchmarking and testing efforts to see the benefits of the various vertical and horizontal scaling improvements.

We recently presented them with the details at KubeCon EU 2024. Here’s the link:
Key Takeaways from Scaling Adobe’s CI/CD Solution to Support 50K Argo CD Apps

We also published a blog with focus on Argo Workflows scalability testing:
Argo Workflows Controller Scalability Testing on Amazon EKS

Challenges / Risks

While this architecture works well for us and provides us with a lot of flexibility, every architecture has some challenges, risks and overheads. Here are a few for FiaB architecture:

Infrastructure cost - There is some extra cost of running each Flexbox due to the additional control plane and the extra capacity buffer per cluster.
Complexity - Architecture becomes more complex because of adding a new Redirector component between GitHub and Flexbox. As a side effect, Redirector becomes a single point of failure across Flexboxes.
Thresholds/Limits - Defining and managing limits for each Flexbox is a challenge.
Support cost - The support team has to understand the role played by Redirector and Relocator when troubleshooting issues.
User experience - If a service gets relocated, some links exposed to clients may change e.g. Argo CD and Argo Workflows URLs, along with Hub cluster information.

Use cases

The Flex-in-a-box architecture opens up a world of possibilities for us in the future. Here are a few use cases:

Dedicated Flexboxes for specific orgs/teams - Some teams have already requested dedicated Flexboxes for their teams to effectively manage the noisy neighbor problem, and also enable efficient chargeback.
- This also allows us to shard the clusters better across Flexboxes.
Domain-specific Flexboxes - Specific Flexboxes could be dedicated to specific domains. For example:
1. Windows - Windows builds are a different beast than builds for Linux. Services workflows needing Windows builds may get routed to a Flexbox with specific tools/components to do a Windows build.
2. Desktop / Mobile - Builds for desktop and mobile apps may need a dedicated build farm. One or more Flexboxes can be designated/dedicated for this.
3. Cloud Infrastructure Provisioning - Cloud infrastructure provisioning requires custom operators and may need a separate Flexbox to ensure that specific custom operators do not adversely affect the performance of existing workloads. We can provide a dedicated box for hosting such workloads.
4. FaaS/Wasm - Building and testing Wasm modules could be very different than building containers. We can provide a dedicated box for hosting FaaS workflows.
Toolchain-specific Flexboxes - Flexboxes could cater to specific toolchains too. For example, if a service is using GitHub Actions (GHA) as the technology for pipeline creation and executions, GHA runners for the same could run in a specific Flexbox.

What’s Next / Future Exploration

Flex-in-a-Box improvements

While this architecture will serve us well in the future, here are some areas of future exploration:

Better sharding - As of now, all remote clusters are registered with Flexbox 1 and Flexbox 2. There is a hidden performance impact on the remote cluster for each Argo CD instance monitoring it. We can improve the sharding to have specific orgs/services on a specific Flexbox. This allows only clusters associated with those services to be registered with Argo CD on that box.
Split Argo CD and Argo Workflows across clusters - One of the challenges with current architecture is that both Argo CD and Argo Workflows are on the same cluster. This can be improved by running Argo CD on a cluster other than the one running Argo Workflows. This separate cluster could be another physical cluster in the same Flexbox, or it could be a virtual cluster (e.g. vCluster) running on the existing physical/host cluster.

Open-sourcing

Alongside the architecture improvements, Adobe is an integral part of CNOE cohort and we are working with a few companies (inside and outside of CNOE) to see how they can use what we have already built, and open-source Flex (or perhaps parts of it to start with).

Unified CI/CD

We have started a new initiative called Unified CI/CD which has the following goals:

Beyond containers - We primarily cater to container based workloads and this initiative helps us serve non-container use cases such as mobile, static website, desktop, serverless etc.
Secure by default - We aim to incorporate all non-negotiable security best practices by default in our pipelines and enforce them.
Sealed Paved Roads - We are working towards providing paved roads/paths for the most common use cases and make them “sealed”, minimizing the need (and making it hard) to deviate from the paved road while ensuring standardization, configurability, security, compliance, and other best practices.
Simplified workflows - Make the workflows simpler to ensure faster onboarding, reduced time for “zero to first commit”, faster “Concept -> Code -> Running state”.

Key Takeaways / Lessons Learnt

There were many key takeaways in this journey and several lessons learnt that can benefit the community. Here are a few:

Developer Experience as a Product - Developer Experience should be treated as a product and non-functional requirements (NFRs, such as scalability, stability, performance etc.) should be given as much importance as functionality.
Plan for future scale - Plan upfront about horizontal and vertical scaling needs, and how a combination of the two can help you achieve your scaling needs. The architecture should be able to evolve to handle future requirements and scale.
Sharding Strategy - Think of a sharding strategy for your services depending on your requirements. For Adobe, specific orgs/services on a specific Flexbox works out well.
Flexbox automation - Automate the creation of “your Flexboxes” from the beginning. This helps bring in predictability and reliability.
Performance Monitoring - Monitor key performance metrics from the beginning to figure when your infrastructure is under stress. We figured it too late and it led to clients facing issues before we could detect them.
K8s Control Plane as the bottleneck - We realized it pretty late that K8s control plane can become a bottleneck and how Argo CD and Argo Workflows can pound the K8s API server. Avoid having Argo CD and Argo Workflows on the same cluster.
Plan for Disasters / Relocation - Disasters will happen. Clusters will go down. Think of how you would deal with it. Can you relocate workloads to another cluster? Another Flexbox? What would be the Developer Experience before/during/after relocation?

Conclusion

A rich and seamless developer experience acts as a developer productivity multiplier and helps any enterprise compete better. Flex-in-a-box architecture provides Adobe with a solid foundation which we can use to scale our current developer experience foundation for future needs. This includes developer experience needs for use cases such as Windows, Serverless, Cloud Infrastructure, Desktop and Mobiles, Static Websites etc.

Discussion

End user members may participate in the discussion thread for this architecture.