Demystifying Deployment of Nephio Workload Clusters on Multi-Cloud

Welcome back to our technical deep-dive series on Nephio! Following our previous discussion on the capabilities of Nephio, this blog post focuses on how we can extend Nephio to deploy workload clusters on multi-cloud. We will discuss in detail about creating and deploying a new workload cluster kpt package for Azure AKS, using cluster API Kubernetes operator.

Nephio’s fundamental elements comprise Nephio controllers and a collection of open-source tools. The primary tools integrated into the framework consist of kpt, porch, gitea, and configsync. Therefore, prior to delving into the deployment of the workload cluster package, we will go through specific details of these core toolchains and their key functionalities.

Diagram 1: Nephio Deployment

Kpt is a tool that simplifies the way we manage and customize Kubernetes configurations. It treats these configurations as data, which means they’re the master plan that defines how things should work in a Kubernetes cluster. kpt organizes these configurations into packages, which are like folders containing all the instructions Kubernetes needs, written in YAML files.

A kpt package is identified by a special file called a Kptfile which holds information about the package, kind of like a table of contents. Just as you can have folders within folders on your computer, kpt packages can contain subpackages, allowing for complex but organized configurations.

Nephio relies on kpt for these capabilities:

• Automation: kpt brings the automation needed for managing configurations at scale.
• Customization: It allows for the necessary customization of configurations, which is a common requirement for real-world deployments. This is also possible with helm like tools, but they use templates and sometimes over parameterization.
• Version Control: kpt uses Git for version control, providing a familiar workflow for managing changes.
• Interoperability: With its function-based approach, kpt ensures that different tools and processes can work together smoothly.

Porch is one of the key components within the Nephio project aimed at simplifying the management of Kubernetes configurations. Its approach to package orchestration underscores Nephio’s intent-driven automation by ensuring that the high-level objectives of network deployments are translated into effective, actionable configurations. It facilitates the creation, tracking, and updating of KRM files as KPT packages. These packages contain configuration data that define how software should be run in a Kubernetes cluster. It also provides a UI experience of WYSIWYG (what you see is what you get) and is the main UI for Nephio in R1 release.

In essence, Porch is designed to handle these major tasks:

• Package Versioning: It keeps track of different versions of configuration packages, allowing for easy updates and rollbacks.
• Repository Management: Porch helps manage repositories where these packages are stored, making it easier to organize and access the configurations needed for deploying network services.
• Package Lifecycle Management: From creating new configurations to proposing changes and publishing final versions, Porch automates and streamlines the entire process.
• Deployment Readiness: It ensures that once configurations are deemed ready, they can be deployed to the actual Kubernetes environments whether they’re in the cloud or at the network edge.

We may need clarification now as to why we need both porch and kpt, so in simple terms kpt is the client-side tool while porch is a server-side tool to handle packages and its lifecycle.

Nephio R1 relies on configsync to implement Gitops capability, but this can be easily replaced with any other Gitops tool. It helps the KRM resources on the cluster always be in sync with the kpt package revision in Git repository. Porch manages the lifecycle of the packages, but it is configsync which applies and actuates the Kubernetes resources. Later in the section, we will detail how argocd can be used as a Gitops tool to sync packages to cluster.

Gitea is the primary git tool that comes with R1 release of Nephio. This is where the repositories are created, which will be registered and managed through porch. There are two types of repositories: a blueprint repository which holds model packages and a deployment repository, which contains package instances. This tool also can be replaced with any other git repository like github.

Nephio’s primary capabilities includes its ability to provision and configure multi-vendor cloud infrastructure as workload clusters. While it natively supports kind and GCP as the designated cluster platforms, it is adaptable to other cloud providers through the installation of the respective custom resource and the creation of the necessary KPT packages. The upcoming sections will guide you through the process of creating the Nephio blueprint upstream package and provisioning of workload cluster downstream package on Azure.

Pre-requisites:

• An active Azure account.
• An Existing Kubernetes clusters.
• Access to a GitHub account for repository management.

In this demo we have used the Nephio’s install script to install Nephio on google cloud VM. This installs a Kubernetes cluster as well as Nephio core components.  If you do not have access to GCP, you can convert any Kubernetes cluster into Nephio system by installing these components separately as shown in the diagram below.

We want to maintain blueprints which are common for all Azure capi clusters, into a single folder. We create an upstream package “cluster-capi-aks” which will contain yaml configurations to create Azure cluster and Kptfile with setters that needs to be updated when the final package is created. nephio-workload-cluster-aks package is the downstream kpt package that will be cloned and deployed as a workload cluster.

High-level overview of the files in kpt package.

The PackageVariant lets you automate the creation and lifecycle management of a specific configuration variant derived from a source package or upstream (cluster-capi-aks in the blueprints-infra-aks repo). Key aspects include:

• Upstream & Downstream: Specifies the source (upstream) of the configuration and where the modified configuration (downstream) will be stored.
• Injectors: Uses the ConfigMap named azure-context to inject specific Azure configuration details into the package.
• Pipeline Mutators: Defines a sequence of KRM functions (set-annotations, apply-replacements, apply-setters) for transforming the package. These functions modify the package based on the provided configuration data.

The Azure-context ConfigMap includes essential Azure-related configuration data like subscription ID, client ID, tenant ID, and more. Marked as required for config injection (kpt.dev/config-injection), this data is crucial for tailoring the package to a specific Azure environment.

This setters ConfigMap holds key-value pairs for various configuration settings, allowing for dynamic modification of package resources. This includes subscription ID, client ID, tenant ID and all other environment variables that are unique to the deployment and need to be replaced inside the Azure cluster configuration using the setters function.

The ApplyReplacements configuration specifies how to propagate certain values throughout the package. It ensures consistency across various components by dynamically updating fields based on the values defined in azure-context and setters ConfigMap

The Kptfile is central to managing the package lifecycle with kpt. It describes the package (nephio-workload-cluster-aks) and defines the pipeline of functions to be applied to the package, ensuring the desired transformations are executed.

The user-defined values are configured in the Azure context file which will hold data, as we configure the azure-context.yaml within git, the “annotation config.kubernetes.io/local-config is set to false”.

To integrate the above blueprint repository with Porch for cloning and deployment, apply the following YAML to create the repository and the associated secret:

Once the repo is registered, you should see all the revisions and tags available as remote packages.

We need to create package revisions to allow configsync to execute PackageVariant definitions on the management server.

The first step is to clone the nephio-workload-cluster-aks package to the target repository, then propose and approve. It is not required to modify anything locally as all rendering will be done by package variant based on the pipeline setup.

The usual flow of a package is depicted below.

The first time when you register the repo, only the main branch and main revision will be visible.  When creating package revisions for the first time use the clone command, and for any subsequent revision, use the copy command.

In this demo, our target repository is “mgmt”, and we want the package to be executed on the mgmt cluster. Once the package is approved in porch, the configsync, which has 1-1 mapping with mgmt repository, will execute the packages as a new revision is made available.

This package revision will clone the upstream package and perform kpt fn rendering and create azure workload cluster.

We can use PackageVariantSet to create multiple clusters at the same time based on selectors if required. It has a similar template to PackageVariant actuating upstream packages.

It is important to mention that there could be additional NF related CRDs and network configurations required on the workload cluster to deploy free5gcore and OAI RAN, which is not covered in this blog.

We can also use argocd as a Gitops tool in Nephio to sync deployment repositories to workload clusters. But it is important to note that argocd doesn’t support full features of kpt packages like functions.
Follow these steps to install and configure argocd on Kubernetes clusters.

There are a few steps involved in creating a sync between the workload repo and the newly created workload cluster.

1. Get the cluster kubeconfig. In this demo, we created an Azure cluster, so we used azurecli to get kubeconfig saved.
2. Add the clusters to argocd
3. Add the repo to argocd. For demo we placed a simple nginx deployment to the repo.
4. Create application which defines the source repo and destination kubernetes cluster. There is a more detailed configuration with regards to how you want to perform the sync. In this demo we create a simple config without the autosync option enabled.

Once the application is created. Click the sync and perform the sync. We can enable the autosync either via declarative configuration or through UI.