From ab4692cd54bd0838f501311eca1ff506625b4d1d Mon Sep 17 00:00:00 2001 From: Stefan Prodan Date: Sat, 27 Apr 2024 12:27:15 +0300 Subject: [PATCH] [RFC] Flux Bootstrap for OCI-compliant Container Registries Signed-off-by: Stefan Prodan --- rfcs/000X-flux-bootstrap-oci/README.md | 319 +++++++++++++++++++++++++ 1 file changed, 319 insertions(+) create mode 100644 rfcs/000X-flux-bootstrap-oci/README.md diff --git a/rfcs/000X-flux-bootstrap-oci/README.md b/rfcs/000X-flux-bootstrap-oci/README.md new file mode 100644 index 00000000..5223835f --- /dev/null +++ b/rfcs/000X-flux-bootstrap-oci/README.md @@ -0,0 +1,319 @@ +# [RFC] Flux Bootstrap for OCI-compliant Container Registries + +**Status:** provisional + +**Creation date:** 2024-04-27 + +**Last update:** 2024-04-27 + +## Summary + +Flux should allow a Git-less bootstrap procedure where the cluster desired state is stored in OCI artifacts. + +On the client-side, the Flux CLI should offer a command for packaging its own Kubernetes manifests into +an OCI artifact and pushing the artifact to a container registry. + +On the server-side, the Flux controllers should be configured to self-update from the registry +and reconcile the cluster state from OCI artifacts stored in the same or a different registry. + +## Motivation + +Given that OCI registries are evolving into a generic artifact storage solution, +we should allow Flux users who don't want to run a Git server as part of their +production infrastructure to bootstrap and manage their Kubernetes clusters using OCI artifacts. + +To decouple the clusters reconciliation from the Git repositories, Flux allows packaging and publishing +the Kubernetes manifests stored in Git to an OCI registry by running the `flux push artifact` +command in CI pipelines. + +### Goals + +- Add support to the Flux CLI for bootstrapping with a container registry as the source of truth. +- Make it easy for users to switch from Git repositories to OCI repositories. + +### Non-Goals + +- Automate the migration of Flux manifests from a Git bootstrap repository to OCI. + +## Proposal + +Implement the `flux bootstrap oci` command with the following specifications: + +```shell +flux bootstrap oci \ +--url=/: \ +--username= \ +--password= \ +--kustomization= \ +--cluster-url=/: \ +--cluster-path= +``` + +The Terraform/OpenTofu counterpart is the `flux_bootstrap_oci` provider that exposes +the same configuration options as the CLI. + +The bootstrap operations are split into two phases: + +- Install and self-update configuration for the Flux components. +- Cluster state reconciliation configuration. + +### Install and self-update configuration + +The command performs the following steps based on the `url`, `username`, +`password` and `kustomization` arguments: + +1. Logs in to the OCI registry using the provided credentials. +2. Generates an OCI artifact from the Flux components manifests and the `kustomization.yaml` file. +3. Applies the Flux components manifests along with their customisations to the cluster. +4. Pushes the OCI artifact to the container registry using the specified tag. +5. Generates an image pull secret, an OCIRepository that points to the OCI artifact and + a Flux Kustomization object that reconciles the OCI artifact contents. +6. Applies the image pull secret, OCIRepository and Flux Kustomization to the cluster. + +Artifacts pushed to the registry: +- `/:` (immutable artifact) +- `/:` (tag pointing to the immutable artifact) + +Objects created by the command in the `flux-system` namespace: +- `flux-components` Secret +- `flux-components` OCIRepository +- `flux-components` Kustomization + +### Cluster state reconciliation configuration + +After the OCIRepository and Flux Kustomization called `flux` become ready, the command +continues with the following steps: + +1. Logs in to the OCI registry where the cluster artifacts are stored using the provided credentials. +2. If the cluster OCI artifact is not found, an empty artifact is created + and pushed to the registry using the provided tag. +3. Generates an image pull secret, an OCIRepository and a Flux Kustomization object + that reconciles the cluster OCI artifact contents. +4. Applies the image pull secret, OCIRepository and Flux Kustomization to the cluster. + +Objects created by the command in the `flux-system` namespace: +- `flux-system` Secret +- `flux-system` OCIRepository +- `flux-system` Kustomization + +If the cluster registry is the same as the Flux components registry, the command could reuse the +`flux-components` image pull secret. + +### Registry authentication + +The `flux bootstrap oci` command supports the following authentication methods: + +- Basic authentication with `--username` and `--password`. The credentials are stored in a Kubernetes Secret. +- OIDC authentication with `--provider=`. No credentials are stored in the cluster, source-controller + will use Kubernetes Workload Identity to authenticate to the registry. + +To avoid passing the credentials as CLI flags, the password can be read from the standard input, e.g.: +`echo | flux bootstrap oci` or using an environment variable `OCI_PASSWORD`. + +If the registry is self-hosted and uses a self-signed TLS certificate, +the root CA certificate can be provided with the `--ca-file` flag. + +If the registry is exposed on HTTP and not HTTPS, the `--allow-insecure-http` +flag can be used to force non-TLS connections. + +### Signing and verification + +The `flux bootstrap oci` command supports the following signing and verification methods: + +- Cosign +- Notation + +TODO: Add more details about the signing and verification methods, flags and options. + +### User Stories + +#### Story 1 + +> As a platform operator I want to bootstrap a Kubernetes cluster with Flux +> using OCI artifacts stored in a container registry. + +The following example demonstrates how to bootstrap a Flux instance using GitHub Container Registry +as the OCI registry for Flux components and the cluster state. + +```shell +flux bootstrap oci \ +--url=ghcr.io/stefanprodan/flux-manifests:production \ +--username= \ +--password= \ +--kustomization=flux-manifests/kustomization.yaml \ +--cluster-url=ghcr.io/stefanprodan/fleet-manifests:production \ +--cluster-username= \ +--cluster-password= \ +--cluster-path=clusters/production +``` + +Generated OCI artifacts: + +- `ghcr.io/stefanprodan/flux-manifests:88b028f` +- `ghcr.io/stefanprodan/flux-manifests:production` +- `ghcr.io/stefanprodan/fleet-manifests:6f7a258` +- `ghcr.io/stefanprodan/fleet-manifests:production` + +Objects created in the `flux-system` namespace: + +Flux components reconciliation: + +```yaml +apiVersion: source.toolkit.fluxcd.io/v1beta2 +kind: OCIRepository +metadata: + name: flux-components + namespace: flux-system +spec: + interval: 1m + url: oci://ghcr.io/stefanprodan/flux-manifests + ref: + tag: production + secretRef: + name: flux-components +--- +apiVersion: kustomize.toolkit.fluxcd.io/v1 +kind: Kustomization +metadata: + name: flux-components + namespace: flux-system +spec: + interval: 1h + retryInterval: 5m + sourceRef: + kind: OCIRepository + name: flux-components + path: ./ + prune: true +``` + +Cluster state reconciliation: + +```yaml +apiVersion: source.toolkit.fluxcd.io/v1beta2 +kind: OCIRepository +metadata: + name: flux-system + namespace: flux-system +spec: + interval: 1m + url: oci://ghcr.io/stefanprodan/fleet-manifests + ref: + tag: production + secretRef: + name: flux-system +--- +apiVersion: kustomize.toolkit.fluxcd.io/v1 +kind: Kustomization +metadata: + name: flux-system + namespace: flux-system +spec: + interval: 1h + retryInterval: 5m + sourceRef: + kind: OCIRepository + name: flux-system + path: clusters/production + prune: true +``` + +#### Story 2 + +> As a platform operator I want to sync the cluster state with the fleet Git repository. + +Push changes from the fleet Git repository to the container registry: + +```shell +# clone the fleet Git repository +git clone https://github.com/stefanprodan/fleet.git +cd fleet +git switch main + +# push the contents the fleet OCI repository and tag it with the commit short SHA +flux push artifact oci://ghcr.io/stefanprodan/fleet-manifests:$(git rev-parse --short HEAD) \ +--path="./" \ +--source="$(git config --get remote.origin.url)" \ +--revision="$(git branch --show-current)@sha1:$(git rev-parse HEAD)" + +# tag the new version for production +flux tag artifact oci://ghcr.io/stefanprodan/fleet-manifests:$(git rev-parse --short HEAD) \ +--tag=production +``` + +This operation can be automated using the Flux GitHub Action. + +The Git repository structure would be similar to the +[flux2-kustomize-helm-example](https://github.com/fluxcd/flux2-kustomize-helm-example) with the following changes: + +- The `clusters/production/flux-system` directory is no more. +- The Flux Kustomization objects defined in the `clusters/production` directory, such as + `infrastructure.yaml` and `apps.yaml`, have the `.spec.sourceRef` set to + `kind: OCIRepository` and `name: flux-system`. + +#### Story 3 + +> As a platform operator I want to update the Flux controllers on my production cluster +> from CI without access to the Kubernetes API. + +Download the latest CLI version and update Flux directly in the registry, without rerunning bootstrap: + +```shell +# pull the latest manifests from the registry +flux pull artifact oci://ghcr.io/stefanprodan/flux-manifests:production \ +--output=./flux-manifests + +# update the Flux components manifests +flux install --export > ./flux-manifests/flux-system/gotk-components.yaml + +# calculate the checksum of the manifests +checksum=$(grep -ar -e . ./flux-manifests/ | shasum | cut -c-16) + +# extract the Flux version and commit +flux_version=$(flux version --client | awk '{print $2}') +flux_commit=$(go version -m $(which flux) | grep vcs.revisio | awk -F= '{print $NF}') + +# push the updated manifests to the registry using the checksum as tag +flux push artifact oci://ghcr.io/stefanprodan/flux-manifests:${checksum} \ +--path="./flux-manifests" \ +--source="https://github.com/fluxcd/flux2" \ +--revision="${flux_version}@sha1:${flux_commit}" + +# tag the new version for production +flux tag artifact oci://ghcr.io/stefanprodan/flux-manifests:${checksum} \ +--tag=production +``` + +This operation could be simplified by implementing a dedicated CLI command and/or GitHub Action. + +#### Story 4 + +> As a platform operator I want to update the registry credentials on my clusters. + +To rotate the registry credentials, generate a new GitHub token and overwrite the image pull secret: + +```shell +flux create secret oci flux-system \ +--url=ghcr.io \ +--username= \ +--password= +``` + +Another option is to rerun the bootstrap command with the new credentials. + +## Design Details + +The bootstrap feature will be implemented as a Go package under `fluxcd/flux2/pkg/bootstrap/oci` +using the [fluxcd/pkg/oci](https://github.com/fluxcd/pkg/tree/main/oci) +library for OCI operations such as auth, push, pull, tag, etc. + +Both the Flux CLI and the Terraform/OpenTofu provider will use the `fluxcd/flux2/pkg/bootstrap/oci` package +and expose the same configuration options. + +### Enabling the feature + +The feature is enabled by default. + +## Implementation History + +* NONE \ No newline at end of file