flux2/rfcs/000X-flux-bootstrap-oci/README.md

# [RFC] Flux Bootstrap for OCI-compliant Container Registries

**Status:** provisional

**Creation date:** 2024-04-27

**Last update:** 2024-04-27

## Summary

Flux should allow a Git-less bootstrap procedure where the cluster desired state is stored in OCI artifacts.

On the client-side, the Flux CLI should offer a command for packaging its own Kubernetes manifests into
an OCI artifact and pushing the artifact to a container registry.

On the server-side, the Flux controllers should be configured to self-update from the registry
and reconcile the cluster state from OCI artifacts stored in the same or a different registry.

## Motivation

After the implementation of [RFC-0003](../0003-kubernetes-oci/README.md) in 2022 and the introduction
of the `OCIRepository` source, we had a recurring ask from users about improving the UX of running
Flux fully decoupled from Git.

Given that OCI registries are evolving into a generic artifact storage solution,
we should assist Flux users who don't want to depend on a Git (for any reason,
including auth and SSH key management) in their production infrastructure to
bootstrap and manage their Kubernetes clusters using OCI artifacts.

To decouple the clusters reconciliation from the Git repositories, Flux allows packaging and publishing
the Kubernetes manifests stored in Git to an OCI registry by running the `flux push artifact`
command in CI pipelines.

### Goals

- Add support to the Flux CLI for bootstrapping with a container registry as the source of truth.
- Make it easy for users to switch from Git repositories to OCI repositories.

### Non-Goals

- Automate the migration of Flux manifests from a Git bootstrap repository to OCI.

## Proposal

Implement the `flux bootstrap oci` command with the following specifications:

```shell
flux bootstrap oci \
--url=oci://<registry-url>/<flux-manifests>:<tag> \
--username=<registry-username> \
--password=<registry-password> \
--kustomization=<local/path/to/kustomization.yaml> \
--cluster-url=oci://<registry-url>/<fleet-manifests>:<tag> \
--cluster-path=<path/inside/oci/artifact>
```

The Terraform/OpenTofu counterpart is the `flux_bootstrap_oci` provider that exposes
the same configuration options as the CLI.

The bootstrap operations are split into two phases:

- Install and self-update configuration for the Flux components.
- Cluster state reconciliation configuration.

### Install and self-update configuration

The command performs the following steps based on the `url`, `username`,
`password` and `kustomization` arguments:

1. Logs in to the OCI registry using the provided credentials.
2. Generates an OCI artifact from the Flux components manifests and the `kustomization.yaml` file.
3. Applies the Flux components manifests along with their customisations to the cluster.
4. Pushes the OCI artifact to the container registry using the specified tag.
5. Generates an image pull secret, an OCIRepository that points to the OCI artifact and
   a Flux Kustomization object that reconciles the OCI artifact contents.
6. Applies the image pull secret, OCIRepository and Flux Kustomization to the cluster.

Note that the creation of the image pull secret is skipped when
[Kubernetes Workload Identity](#story-2) is used for authentication to the container registry.

Artifacts pushed to the registry:
- `<registry-url>/<flux-manifests>:<checksum>` (immutable artifact)
- `<registry-url>/<flux-manifests>:<tag>` (tag pointing to the immutable artifact)

Objects created by the command in the `flux-system` namespace:
- `flux-components` Secret
- `flux-components` OCIRepository
- `flux-components` Kustomization

### Cluster state reconciliation configuration

After the OCIRepository and Flux Kustomization called `flux` become ready, the command
continues with the following steps:

1. Logs in to the OCI registry where the cluster artifacts are stored using the provided credentials.
2. If the cluster OCI artifact is not found, an empty artifact is created
   and pushed to the registry using the provided tag.
3. Generates an image pull secret, an OCIRepository and a Flux Kustomization object
   that reconciles the cluster OCI artifact contents.
4. Applies the image pull secret, OCIRepository and Flux Kustomization to the cluster.

Note that the creation of the image pull secret is skipped when
[Kubernetes Workload Identity](#story-2) is used for authentication to the container registry.

Objects created by the command in the `flux-system` namespace:
- `flux-system` Secret
- `flux-system` OCIRepository
- `flux-system` Kustomization

If the cluster registry is the same as the Flux components registry, the command could reuse the
`flux-components` image pull secret.

### Registry authentication

The `flux bootstrap oci` command supports the following authentication methods:

- Basic authentication with `--username` and `--password`. The credentials are stored in a Kubernetes Secret.
- OIDC authentication with `--provider=<aws|azure|gcp>`. No credentials are stored in the cluster, source-controller
  will use Kubernetes Workload Identity to authenticate to the registry.

To avoid passing the credentials as CLI flags, the password can be read from the standard input, e.g.:
`echo <password> | flux bootstrap oci` or using an environment variable `OCI_PASSWORD`.

If the registry is self-hosted and uses a self-signed TLS certificate,
the root CA certificate can be provided with the `--ca-file` flag.

If the registry is exposed on HTTP and not HTTPS, the `--allow-insecure-http`
flag can be used to force non-TLS connections.

### Signing and verification

The `flux bootstrap oci` command supports the following signing and verification methods:

- Cosign
- Notation

TODO: Add more details about the signing and verification methods, flags and options.

### User Stories

#### Story 1

> As a platform operator I want to bootstrap a Kubernetes cluster with Flux
> using OCI artifacts stored in a container registry.

The following example demonstrates how to bootstrap a Flux instance using GitHub Container Registry
as the OCI registry for Flux components and the cluster state.

```shell
flux bootstrap oci \
--url=oci://ghcr.io/stefanprodan/flux-manifests:production \
--username=<ghcr-username> \
--password=<ghcr-token> \
--kustomization=flux-manifests/kustomization.yaml \
--cluster-url=oci://ghcr.io/stefanprodan/fleet-manifests:production \
--cluster-username=<ghcr-username> \
--cluster-password=<ghcr-token> \
--cluster-path=clusters/production
```

Generated OCI artifacts:

- `ghcr.io/stefanprodan/flux-manifests:88b028f`
- `ghcr.io/stefanprodan/flux-manifests:production`
- `ghcr.io/stefanprodan/fleet-manifests:6f7a258`
- `ghcr.io/stefanprodan/fleet-manifests:production`

Objects created in the `flux-system` namespace:

Flux components reconciliation:

```yaml
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: OCIRepository
metadata:
  name: flux-components
  namespace: flux-system
spec:
  interval: 1m
  url: oci://ghcr.io/stefanprodan/flux-manifests
  ref:
    tag: production
  secretRef:
    name: flux-components
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: flux-components
  namespace: flux-system
spec:
    interval: 1h
    retryInterval: 5m
    sourceRef:
        kind: OCIRepository
        name: flux-components
    path: ./
    prune: true
```

Cluster state reconciliation:

```yaml
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: OCIRepository
metadata:
  name: flux-system
  namespace: flux-system
spec:
   interval: 1m
   url: oci://ghcr.io/stefanprodan/fleet-manifests
   ref:
    tag: production
   secretRef:
    name: flux-system
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: flux-system
  namespace: flux-system
spec:
    interval: 1h
    retryInterval: 5m
    sourceRef:
        kind: OCIRepository
        name: flux-system
    path: clusters/production
    prune: true
```

#### Story 2

> As a platform operator I want to bootstrap an EKS cluster with Flux
> using OCI artifacts stored in ECR.

The following example demonstrates how to bootstrap a Flux instance using ECR using IAM auth.
Assuming the EKS nodes have read-only access to ECR and the bastion host where
the Flux CLI is running has read and write access to ECR:

```shell
flux bootstrap oci \
--provider=aws \
--url=oci://aws_account_id.dkr.ecr.us-west-2.amazonaws.com/flux-manifests:production \
--kustomization=flux-manifests/kustomization.yaml \
--cluster-url=oci://aws_account_id.dkr.ecr.us-west-2.amazonaws.com/fleet-manifests:production \
--cluster-path=clusters/production
```

Note that when using Kubernetes Workload Identity instead of the worker node IAM role,
the `kustomization.yaml` must contain patches for the source-controller Service Account
as described [here](https://fluxcd.io/flux/installation/configuration/workload-identity/).

#### Story 3

> As a platform operator I want to sync the cluster state with the fleet Git repository.

Push changes from the fleet Git repository to the container registry:

```shell
# clone the fleet Git repository
git clone https://github.com/stefanprodan/fleet.git
cd fleet
git switch main

# push the contents the fleet OCI repository and tag it with the commit short SHA
flux push artifact oci://ghcr.io/stefanprodan/fleet-manifests:$(git rev-parse --short HEAD) \
--path="./" \
--source="$(git config --get remote.origin.url)" \
--revision="$(git branch --show-current)@sha1:$(git rev-parse HEAD)"

# tag the new version for production
flux tag artifact oci://ghcr.io/stefanprodan/fleet-manifests:$(git rev-parse --short HEAD) \
--tag=production
```

This operation can be automated using the Flux GitHub Action.

The Git repository structure would be similar to the
[flux2-kustomize-helm-example](https://github.com/fluxcd/flux2-kustomize-helm-example) with the following changes:

- The `clusters/production/flux-system` directory is no more.
- The Flux Kustomization objects defined in the `clusters/production` directory, such as
  `infrastructure.yaml` and `apps.yaml`, have the `.spec.sourceRef` set to
  `kind: OCIRepository` and  `name: flux-system`.

#### Story 4

> As a platform operator I want to update the Flux controllers on my production cluster
> from CI without access to the Kubernetes API.

Download the latest CLI version and update Flux directly in the registry, without rerunning bootstrap:

```shell
# pull the latest manifests from the registry
flux pull artifact oci://ghcr.io/stefanprodan/flux-manifests:production \
--output=./flux-manifests

# update the Flux components manifests
flux install --export > ./flux-manifests/flux-system/gotk-components.yaml

# calculate the checksum of the manifests
checksum=$(grep -ar -e . ./flux-manifests/ | shasum | cut -c-16)

# extract the Flux version and commit
flux_version=$(flux version --client | awk '{print $2}')
flux_commit=$(go version -m  $(which flux) | grep vcs.revisio | awk -F= '{print $NF}')

# push the updated manifests to the registry using the checksum as tag
flux push artifact oci://ghcr.io/stefanprodan/flux-manifests:${checksum} \
--path="./flux-manifests" \
--source="https://github.com/fluxcd/flux2" \
--revision="${flux_version}@sha1:${flux_commit}"

# tag the new version for production
flux tag artifact oci://ghcr.io/stefanprodan/flux-manifests:${checksum} \
--tag=production
```

This operation could be simplified by implementing a dedicated CLI command and/or GitHub Action.

#### Story 5

> As a platform operator I want to update the registry credentials on my clusters.

To rotate the registry credentials, generate a new GitHub token and overwrite the image pull secret:

```shell
flux create secret oci flux-system \
--url=ghcr.io \
--username=<ghcr-username> \
--password=<ghcr-token>
```

Another option is to rerun the bootstrap command with the new credentials.

## Design Details

The bootstrap feature will be implemented as a Go package under `fluxcd/flux2/pkg/bootstrap/oci`
using the [fluxcd/pkg/oci](https://github.com/fluxcd/pkg/tree/main/oci)
library for OCI operations such as auth, push, pull, tag, etc.

Both the Flux CLI and the Terraform/OpenTofu provider will use the `fluxcd/flux2/pkg/bootstrap/oci` package
and expose the same configuration options.

### Enabling the feature

The feature is enabled by default.

## Implementation History

* NONE