mirror of https://github.com/fluxcd/flux2.git
You cannot select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
319 lines
10 KiB
Markdown
319 lines
10 KiB
Markdown
11 months ago
|
# [RFC] Flux Bootstrap for OCI-compliant Container Registries
|
||
|
|
||
|
**Status:** provisional
|
||
|
|
||
|
**Creation date:** 2024-04-27
|
||
|
|
||
|
**Last update:** 2024-04-27
|
||
|
|
||
|
## Summary
|
||
|
|
||
|
Flux should allow a Git-less bootstrap procedure where the cluster desired state is stored in OCI artifacts.
|
||
|
|
||
|
On the client-side, the Flux CLI should offer a command for packaging its own Kubernetes manifests into
|
||
|
an OCI artifact and pushing the artifact to a container registry.
|
||
|
|
||
|
On the server-side, the Flux controllers should be configured to self-update from the registry
|
||
|
and reconcile the cluster state from OCI artifacts stored in the same or a different registry.
|
||
|
|
||
|
## Motivation
|
||
|
|
||
|
Given that OCI registries are evolving into a generic artifact storage solution,
|
||
|
we should allow Flux users who don't want to run a Git server as part of their
|
||
|
production infrastructure to bootstrap and manage their Kubernetes clusters using OCI artifacts.
|
||
|
|
||
|
To decouple the clusters reconciliation from the Git repositories, Flux allows packaging and publishing
|
||
|
the Kubernetes manifests stored in Git to an OCI registry by running the `flux push artifact`
|
||
|
command in CI pipelines.
|
||
|
|
||
|
### Goals
|
||
|
|
||
|
- Add support to the Flux CLI for bootstrapping with a container registry as the source of truth.
|
||
|
- Make it easy for users to switch from Git repositories to OCI repositories.
|
||
|
|
||
|
### Non-Goals
|
||
|
|
||
|
- Automate the migration of Flux manifests from a Git bootstrap repository to OCI.
|
||
|
|
||
|
## Proposal
|
||
|
|
||
|
Implement the `flux bootstrap oci` command with the following specifications:
|
||
|
|
||
|
```shell
|
||
|
flux bootstrap oci \
|
||
|
--url=<registry-url>/<flux-manifests>:<tag> \
|
||
|
--username=<registry-username> \
|
||
|
--password=<registry-password> \
|
||
|
--kustomization=<local/path/to/kustomization.yaml> \
|
||
|
--cluster-url=<registry-url>/<fleet-manifests>:<tag> \
|
||
|
--cluster-path=<path/inside/oci/artifact>
|
||
|
```
|
||
|
|
||
|
The Terraform/OpenTofu counterpart is the `flux_bootstrap_oci` provider that exposes
|
||
|
the same configuration options as the CLI.
|
||
|
|
||
|
The bootstrap operations are split into two phases:
|
||
|
|
||
|
- Install and self-update configuration for the Flux components.
|
||
|
- Cluster state reconciliation configuration.
|
||
|
|
||
|
### Install and self-update configuration
|
||
|
|
||
|
The command performs the following steps based on the `url`, `username`,
|
||
|
`password` and `kustomization` arguments:
|
||
|
|
||
|
1. Logs in to the OCI registry using the provided credentials.
|
||
|
2. Generates an OCI artifact from the Flux components manifests and the `kustomization.yaml` file.
|
||
|
3. Applies the Flux components manifests along with their customisations to the cluster.
|
||
|
4. Pushes the OCI artifact to the container registry using the specified tag.
|
||
|
5. Generates an image pull secret, an OCIRepository that points to the OCI artifact and
|
||
|
a Flux Kustomization object that reconciles the OCI artifact contents.
|
||
|
6. Applies the image pull secret, OCIRepository and Flux Kustomization to the cluster.
|
||
|
|
||
|
Artifacts pushed to the registry:
|
||
|
- `<registry-url>/<flux-manifests>:<checksum>` (immutable artifact)
|
||
|
- `<registry-url>/<flux-manifests>:<tag>` (tag pointing to the immutable artifact)
|
||
|
|
||
|
Objects created by the command in the `flux-system` namespace:
|
||
|
- `flux-components` Secret
|
||
|
- `flux-components` OCIRepository
|
||
|
- `flux-components` Kustomization
|
||
|
|
||
|
### Cluster state reconciliation configuration
|
||
|
|
||
|
After the OCIRepository and Flux Kustomization called `flux` become ready, the command
|
||
|
continues with the following steps:
|
||
|
|
||
|
1. Logs in to the OCI registry where the cluster artifacts are stored using the provided credentials.
|
||
|
2. If the cluster OCI artifact is not found, an empty artifact is created
|
||
|
and pushed to the registry using the provided tag.
|
||
|
3. Generates an image pull secret, an OCIRepository and a Flux Kustomization object
|
||
|
that reconciles the cluster OCI artifact contents.
|
||
|
4. Applies the image pull secret, OCIRepository and Flux Kustomization to the cluster.
|
||
|
|
||
|
Objects created by the command in the `flux-system` namespace:
|
||
|
- `flux-system` Secret
|
||
|
- `flux-system` OCIRepository
|
||
|
- `flux-system` Kustomization
|
||
|
|
||
|
If the cluster registry is the same as the Flux components registry, the command could reuse the
|
||
|
`flux-components` image pull secret.
|
||
|
|
||
|
### Registry authentication
|
||
|
|
||
|
The `flux bootstrap oci` command supports the following authentication methods:
|
||
|
|
||
|
- Basic authentication with `--username` and `--password`. The credentials are stored in a Kubernetes Secret.
|
||
|
- OIDC authentication with `--provider=<aws|azure|gcp>`. No credentials are stored in the cluster, source-controller
|
||
|
will use Kubernetes Workload Identity to authenticate to the registry.
|
||
|
|
||
|
To avoid passing the credentials as CLI flags, the password can be read from the standard input, e.g.:
|
||
|
`echo <password> | flux bootstrap oci` or using an environment variable `OCI_PASSWORD`.
|
||
|
|
||
|
If the registry is self-hosted and uses a self-signed TLS certificate,
|
||
|
the root CA certificate can be provided with the `--ca-file` flag.
|
||
|
|
||
|
If the registry is exposed on HTTP and not HTTPS, the `--allow-insecure-http`
|
||
|
flag can be used to force non-TLS connections.
|
||
|
|
||
|
### Signing and verification
|
||
|
|
||
|
The `flux bootstrap oci` command supports the following signing and verification methods:
|
||
|
|
||
|
- Cosign
|
||
|
- Notation
|
||
|
|
||
|
TODO: Add more details about the signing and verification methods, flags and options.
|
||
|
|
||
|
### User Stories
|
||
|
|
||
|
#### Story 1
|
||
|
|
||
|
> As a platform operator I want to bootstrap a Kubernetes cluster with Flux
|
||
|
> using OCI artifacts stored in a container registry.
|
||
|
|
||
|
The following example demonstrates how to bootstrap a Flux instance using GitHub Container Registry
|
||
|
as the OCI registry for Flux components and the cluster state.
|
||
|
|
||
|
```shell
|
||
|
flux bootstrap oci \
|
||
|
--url=ghcr.io/stefanprodan/flux-manifests:production \
|
||
|
--username=<ghcr-username> \
|
||
|
--password=<ghcr-token> \
|
||
|
--kustomization=flux-manifests/kustomization.yaml \
|
||
|
--cluster-url=ghcr.io/stefanprodan/fleet-manifests:production \
|
||
|
--cluster-username=<ghcr-username> \
|
||
|
--cluster-password=<ghcr-token> \
|
||
|
--cluster-path=clusters/production
|
||
|
```
|
||
|
|
||
|
Generated OCI artifacts:
|
||
|
|
||
|
- `ghcr.io/stefanprodan/flux-manifests:88b028f`
|
||
|
- `ghcr.io/stefanprodan/flux-manifests:production`
|
||
|
- `ghcr.io/stefanprodan/fleet-manifests:6f7a258`
|
||
|
- `ghcr.io/stefanprodan/fleet-manifests:production`
|
||
|
|
||
|
Objects created in the `flux-system` namespace:
|
||
|
|
||
|
Flux components reconciliation:
|
||
|
|
||
|
```yaml
|
||
|
apiVersion: source.toolkit.fluxcd.io/v1beta2
|
||
|
kind: OCIRepository
|
||
|
metadata:
|
||
|
name: flux-components
|
||
|
namespace: flux-system
|
||
|
spec:
|
||
|
interval: 1m
|
||
|
url: oci://ghcr.io/stefanprodan/flux-manifests
|
||
|
ref:
|
||
|
tag: production
|
||
|
secretRef:
|
||
|
name: flux-components
|
||
|
---
|
||
|
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||
|
kind: Kustomization
|
||
|
metadata:
|
||
|
name: flux-components
|
||
|
namespace: flux-system
|
||
|
spec:
|
||
|
interval: 1h
|
||
|
retryInterval: 5m
|
||
|
sourceRef:
|
||
|
kind: OCIRepository
|
||
|
name: flux-components
|
||
|
path: ./
|
||
|
prune: true
|
||
|
```
|
||
|
|
||
|
Cluster state reconciliation:
|
||
|
|
||
|
```yaml
|
||
|
apiVersion: source.toolkit.fluxcd.io/v1beta2
|
||
|
kind: OCIRepository
|
||
|
metadata:
|
||
|
name: flux-system
|
||
|
namespace: flux-system
|
||
|
spec:
|
||
|
interval: 1m
|
||
|
url: oci://ghcr.io/stefanprodan/fleet-manifests
|
||
|
ref:
|
||
|
tag: production
|
||
|
secretRef:
|
||
|
name: flux-system
|
||
|
---
|
||
|
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||
|
kind: Kustomization
|
||
|
metadata:
|
||
|
name: flux-system
|
||
|
namespace: flux-system
|
||
|
spec:
|
||
|
interval: 1h
|
||
|
retryInterval: 5m
|
||
|
sourceRef:
|
||
|
kind: OCIRepository
|
||
|
name: flux-system
|
||
|
path: clusters/production
|
||
|
prune: true
|
||
|
```
|
||
|
|
||
|
#### Story 2
|
||
|
|
||
|
> As a platform operator I want to sync the cluster state with the fleet Git repository.
|
||
|
|
||
|
Push changes from the fleet Git repository to the container registry:
|
||
|
|
||
|
```shell
|
||
|
# clone the fleet Git repository
|
||
|
git clone https://github.com/stefanprodan/fleet.git
|
||
|
cd fleet
|
||
|
git switch main
|
||
|
|
||
|
# push the contents the fleet OCI repository and tag it with the commit short SHA
|
||
|
flux push artifact oci://ghcr.io/stefanprodan/fleet-manifests:$(git rev-parse --short HEAD) \
|
||
|
--path="./" \
|
||
|
--source="$(git config --get remote.origin.url)" \
|
||
|
--revision="$(git branch --show-current)@sha1:$(git rev-parse HEAD)"
|
||
|
|
||
|
# tag the new version for production
|
||
|
flux tag artifact oci://ghcr.io/stefanprodan/fleet-manifests:$(git rev-parse --short HEAD) \
|
||
|
--tag=production
|
||
|
```
|
||
|
|
||
|
This operation can be automated using the Flux GitHub Action.
|
||
|
|
||
|
The Git repository structure would be similar to the
|
||
|
[flux2-kustomize-helm-example](https://github.com/fluxcd/flux2-kustomize-helm-example) with the following changes:
|
||
|
|
||
|
- The `clusters/production/flux-system` directory is no more.
|
||
|
- The Flux Kustomization objects defined in the `clusters/production` directory, such as
|
||
|
`infrastructure.yaml` and `apps.yaml`, have the `.spec.sourceRef` set to
|
||
|
`kind: OCIRepository` and `name: flux-system`.
|
||
|
|
||
|
#### Story 3
|
||
|
|
||
|
> As a platform operator I want to update the Flux controllers on my production cluster
|
||
|
> from CI without access to the Kubernetes API.
|
||
|
|
||
|
Download the latest CLI version and update Flux directly in the registry, without rerunning bootstrap:
|
||
|
|
||
|
```shell
|
||
|
# pull the latest manifests from the registry
|
||
|
flux pull artifact oci://ghcr.io/stefanprodan/flux-manifests:production \
|
||
|
--output=./flux-manifests
|
||
|
|
||
|
# update the Flux components manifests
|
||
|
flux install --export > ./flux-manifests/flux-system/gotk-components.yaml
|
||
|
|
||
|
# calculate the checksum of the manifests
|
||
|
checksum=$(grep -ar -e . ./flux-manifests/ | shasum | cut -c-16)
|
||
|
|
||
|
# extract the Flux version and commit
|
||
|
flux_version=$(flux version --client | awk '{print $2}')
|
||
|
flux_commit=$(go version -m $(which flux) | grep vcs.revisio | awk -F= '{print $NF}')
|
||
|
|
||
|
# push the updated manifests to the registry using the checksum as tag
|
||
|
flux push artifact oci://ghcr.io/stefanprodan/flux-manifests:${checksum} \
|
||
|
--path="./flux-manifests" \
|
||
|
--source="https://github.com/fluxcd/flux2" \
|
||
|
--revision="${flux_version}@sha1:${flux_commit}"
|
||
|
|
||
|
# tag the new version for production
|
||
|
flux tag artifact oci://ghcr.io/stefanprodan/flux-manifests:${checksum} \
|
||
|
--tag=production
|
||
|
```
|
||
|
|
||
|
This operation could be simplified by implementing a dedicated CLI command and/or GitHub Action.
|
||
|
|
||
|
#### Story 4
|
||
|
|
||
|
> As a platform operator I want to update the registry credentials on my clusters.
|
||
|
|
||
|
To rotate the registry credentials, generate a new GitHub token and overwrite the image pull secret:
|
||
|
|
||
|
```shell
|
||
|
flux create secret oci flux-system \
|
||
|
--url=ghcr.io \
|
||
|
--username=<ghcr-username> \
|
||
|
--password=<ghcr-token>
|
||
|
```
|
||
|
|
||
|
Another option is to rerun the bootstrap command with the new credentials.
|
||
|
|
||
|
## Design Details
|
||
|
|
||
|
The bootstrap feature will be implemented as a Go package under `fluxcd/flux2/pkg/bootstrap/oci`
|
||
|
using the [fluxcd/pkg/oci](https://github.com/fluxcd/pkg/tree/main/oci)
|
||
|
library for OCI operations such as auth, push, pull, tag, etc.
|
||
|
|
||
|
Both the Flux CLI and the Terraform/OpenTofu provider will use the `fluxcd/flux2/pkg/bootstrap/oci` package
|
||
|
and expose the same configuration options.
|
||
|
|
||
|
### Enabling the feature
|
||
|
|
||
|
The feature is enabled by default.
|
||
|
|
||
|
## Implementation History
|
||
|
|
||
|
* NONE
|