flux2/rfcs/0012-external-artifact/README.md

# RFC-0012 External Artifact

**Status:** provisional

**Creation date:** 2025-04-08

**Last update:** 2025-09-03

## Summary

This RFC proposes the introduction of a new API called `ExternalArtifact` that would allow
3rd party controllers to act as a source of truth for the cluster desired state. In effect,
the `ExternalArtifact` API acts as an extension of the existing `source.toolkit.fluxcd.io` APIs
that enables Flux `kustomize-controller` and `helm-controller` to consume artifacts from external
source types that are not natively supported by `source-controller`.

## Motivation

Over the years, we've received requests from users to support other source types besides the
ones natively supported by `source-controller`. For example, users have asked for support of
downloading Kubernetes manifests from GitHub/GitLab releases, Omaha protocol, SFTP protocol,
and other remote storage systems.

Another common request is to run transformations on the artifacts fetched by source-controller.
For example, users want to be able to generate YAML manifests from jsonnet, cue, and other
templating engines before they are consumed by Flux `kustomize-controller`.

In order to support these use cases, we need to define a standard API that allows 3rd party
controllers to expose artifacts in-cluster (in the same way `source-controller` does)
that can be consumed by Flux  `kustomize-controller` and `helm-controller`.

### Goals

Define a standard API for 3rd party controllers to expose artifacts that can be consumed by
Flux controllers in the same way as the existing `source.toolkit.fluxcd.io` APIs.

Allow Flux users to transition from using `source-controller` to using 3rd party source controllers
with minimal changes to their existing `Kustomizations` and `HelmReleases`.

### Non-Goals

Allow arbitrary custom resources to be referenced in Flux `Kustomization` and `HelmRelease` as `sourceRef`.

Extend the Flux controllers permissions to access custom resources that are not part of the
`source.toolkit.fluxcd.io` APIs.

## Proposal

Assuming we have a custom controller called `release-controller` that is responsible for
reconciling `GitHubRelease` custom resources. This controller downloads the Kubernetes
deployment YAML manifests from the GitHub API and stores them in a local file system
as a `tar.gz` file. The `release-controller` then creates an `ExternalArtifact`
custom resource that tells the Flux controllers from where to fetch the artifact.

Every time the `release-controller` reconciles a `GitHubRelease` custom resource,
it updates the `ExternalArtifact` status with the latest artifact information if the
upstream release has changed.

The `release-controller` is responsible for exposing a HTTP endpoint that serves
the artifacts from its own storage. The URL of the `tar.gz` artifact is stored in
the `ExternalArtifact` status and should be accessible from the Flux controllers
running in the cluster.

Example of a generated `ExternalArtifact` custom resource:

```yaml
apiVersion: source.toolkit.fluxcd.io/v1
kind: ExternalArtifact
metadata:
  name: podinfo
  namespace: apps
spec:
  # SourceRef points to the Kubernetes custom resource for
  # which the artifact is generated.
  # +optional
  sourceRef:
    apiVersion: source.example.com/v1alpha1
    kind: GitHubRelease
    name: podinfo
    namespace: apps
status:
  artifact:
    # Digest is the digest of the tar.gz file in the form of '<algorithm>:<checksum>'.
    # The digest is used by the Flux controllers to verify the integrity of the artifact.
    # +required
    digest: sha256:35d47c9db0eee6ffe08a404dfb416bee31b2b79eabc3f2eb26749163ce487f52
    # LastUpdateTime is the timestamp corresponding to the last update of the
    # Artifact in storage.
    # +required
    lastUpdateTime: "2025-03-21T13:37:31Z"
    # Path is the relative file path of the Artifact. It can be used to locate
    # the file in the root of the Artifact storage on the local file system of
    # the controller managing the Source.
    # +required
    path: release/apps/podinfo/6.8.0-b3396ad.tar.gz
    # Revision is a human-readable identifier traceable in the origin source system
    # in the form of '<human-readable-identifier>@<algorithm>:<checksum>'.
    # The revision is used by the Flux controllers to determine if the artifact has changed.
    # +required
    revision: 6.8.0@sha256:35d47c9db0eee6ffe08a404dfb416bee31b2b79eabc3f2eb26749163ce487f52
    # Size is the number of bytes of the tar.gz file.
    # +required
    size: 20914
    # URL is the in-cluster HTTP address of the Artifact as exposed by the controller
    # managing the Source. It can be used to retrieve the Artifact for
    # consumption, e.g. by kustomize-controller applying the Artifact contents.
    # +required
    url: http://release-controller.flux-system.svc.cluster.local./release/apps/podinfo/6.8.0-b3396ad.tar.gz
  conditions:
  - lastTransitionTime: "2025-04-08T09:09:49Z"
    message: stored artifact for release 6.8.0
    observedGeneration: 1
    reason: Succeeded
    status: "True"
    type: Ready
```

Note that the `.status.artifact` is identical to how `source-controller` exposes the
artifact information for `Bucket`, `GitRepository`, and `OCIRepository` custom resources.
This allows the Flux controllers to consume external artifacts with minimal changes.

The `ExternalArtifact` custom resource is referenced by a Flux `Kustomization` as follows:

```yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: podinfo
  namespace: apps
spec:
  interval: 10m
  sourceRef:
    kind: ExternalArtifact
    name: podinfo
  path: "./"
  prune: true
```

Flux `kustomize-controller` will then fetch the artifact from the URL specified in the
`ExternalArtifact` status, verifies the integrity of the artifact using the digest
and applies the contents of the artifact to the cluster.

Like with the existing `source.toolkit.fluxcd.io` APIs, `kustomize-controller` will
watch the `ExternalArtifact` custom resource for changes and will re-apply the
contents of the artifact when the `.status.artifact.revision` changes.

When the `ExternalArtifact` contains a Helm chart, it can be referenced by a Flux `HelmRelease` as follows:

```yaml
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: podinfo
  namespace: apps
spec:
  interval: 10m
  releaseName: podinfo
  chartRef:
    kind: ExternalArtifact
    name: podinfo
  values:
    replicaCount: 2
```

### Security Considerations

With the introduction of the `ExternalArtifact` API, the trust boundary of Flux is extended
to include 3rd party controllers that are capable of creating and managing `ExternalArtifact`
custom resources in the cluster. This means that the security posture of the cluster
is now dependent on the security of these 3rd party controllers.

To mitigate potential security risks, it is recommended to implement the following measures 
when developing 3rd party source controllers:

- **Authentication and Authorization**: Ensure that the controller uses proper authentication
  and authorization mechanisms to interact with upstream sources and avoid embedding sensitive
  information directly in the custom resource specifications. Following source-controller
  best practices for managing credentials is highly recommended: use `serviceAccountName` to
  integrate with Kubernetes Workload Identity for short-lived credentials, use `secretRef` to
  reference long-lived credentials, never cache long-lived credentials on disk or in-memory.
- **TLS Encryption**: Use TLS encryption for all communications between the controller
  and upstream sources to protect sensitive data in transit. Following source-controller
  best practices for TLS is highly recommended: use `certSecretRef` to reference
  custom CA certificates and client certificates, prefer Mutual TLS authentication, never
  allow skipping TLS verification.
- **Provenance and Integrity**: Ensure that the controller verifies the integrity of the
  artifacts it generates and exposes in-cluster. This can be achieved by using checksums
  and digital signatures to validate the authenticity of upstream sources. Following
  source-controller best practices for source integrity is highly recommended:
  verify the provenance of upstream artifacts using Sigstore Cosign or Notary
  Notation signatures, prefer keyless verification using OIDC identity tokens and
  public transparency logs.
- **Access Control**: Implement access control mechanisms to restrict cross-namespace
  generation of `ExternalArtifact` custom resources. Following source-controller
  best practices for access control is highly recommended: expose a `--no-cross-namespace-refs`
  flag to restrict the controller from generating `ExternalArtifact` resources in a different
  namespace than the one where the source custom resource is located. Use Kubernetes owner
  references to establish a clear ownership relationship between the source custom resource
  and the `ExternalArtifact` resource, allowing Kubernetes garbage collection to clean up
  the `ExternalArtifact` when the source resource is deleted.
- **Least Privilege**: Run the controller with the least privilege necessary to perform
  its functions. Following source-controller best practices for least privilege is highly recommended:
  use a dedicated Kubernetes service account with minimal RBAC permissions, avoid running
  the controller as a cluster-admin or with wildcard permissions, conform with the restricted pod security
  standard (e.g., disallow running as root, disallow host network access, read-only rootfs).
- **Artifact persistent storage integrity**: Ensure that the controller can be configured to use
  persistent storage for storing artifacts, to avoid data loss in case of controller restarts
  or failures. Following source-controller best practices for artifact storage is highly recommended:
  at startup, ensure that the artifacts in-storage have not been tampered with by verifying
  the checksums of all stored artifacts against the `ExternalArtifact` digests in the cluster.
- **Artifact access restrictions**: If the controller is deployed outside of flux-system namespace,
  it should include network policies that restrict access to the artifact storage endpoint to only
  kustomize-controller and helm-controller.
  Following source-controller best practices for network policies is highly recommended:
  use Kubernetes NetworkPolicies to restrict ingress and egress traffic to/from the controller pods,
  allowing only necessary communication with upstream sources and trusted consumers.

### User Stories

#### 3rd Party Source Controller

As a 3rd party controller developer, I want to expose artifacts in-cluster that are sourced from `flatcar/nebraska`
that can be consumed by Flux `kustomize-controller` and `helm-controller` so that Flux users can use my controller
as a source of truth for their cluster desired state.

#### Custom Source Transform

As a Flux user, I want to use a custom controller that generates Kubernetes manifests from CUE templates 
which can be consumed by Flux `kustomize-controller`.

#### Policy Enforcement

As a cluster administrator, I want to ensure that only trusted 3rd party controllers
can create and manage `ExternalArtifact` resources in the cluster.

```yaml
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: "trusted-external-artifacts"
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
    - apiGroups:   ["source.toolkit.fluxcd.io"]
      apiVersions: ["v1"]
      operations:  ["CREATE", "UPDATE"]
      resources:   ["externalartifacts"]
  validations:
    # Restrict the sourceRef to only allow trusted APIs
    - expression: >
                object.spec.sourceRef.apiVersion.startsWith('source.example.com')
    # Restrict the sourceRef to only allow trusted Kinds
    - expression: >
        object.spec.sourceRef.kind == 'GitHubRelease' || 
        object.spec.sourceRef.kind == 'GitLabRelease'        
    # Restrict the artifacts to be served only by trusted endpoints within the cluster
    - expression: >
        !has(object.status.artifact) ||
        object.status.artifact.url.startsWith('http://release-controller.flux-system.svc.cluster.local./')        
    # Restrict the artifact operations to trusted service accounts
    - expression: >
                request.userInfo.username == 'system:serviceaccount:flux-system:release-controller'
```

### Alternatives

An alternative to this proposal would be to deploy an OCI registry in-cluster.
The 3rd party controllers would then push the artifacts to the registry
and Flux `kustomize-controller` and `helm-controller` would consume the artifacts
via the `OCIRepository` custom resource.

While this approach is feasible, it requires additional infrastructure and
configuration, which may not be desirable for all users. The `ExternalArtifact` API
provides a simpler and more flexible way to expose artifacts in-cluster without
the need to self-host an OCI registry. In addition, the `ExternalArtifact` API
offers a better user experience by allowing Flux user to trace the origin of reconciled resources
back to the original source via `ExternalArtifact.spec.sourceRef` and `flux trace` command.

## Design Details

The `ExternalArtifact` API will be added to `source-controller/api` Go package.
3rd party controllers will import `github.com/fluxcd/source-controller/api/v1`
to generate valid `ExternalArtifact` custom resources using the controller-runtime client.

The Flux maintainers will develop an SDK for packaging and exposing artifacts
in-cluster using the `ExternalArtifact` API. The SDK will provide helper functions
for generating Flux-compliant artifacts, as well as for storing artifacts in a persistent storage.
The SDK will be published as a Go module under the `github.com/fluxcd/pkg` repository.

The `ExternalArtifact` CRD will be bundled with the `source-controller` CRDs manifests which
are part of the standard Flux distribution. This means that users will not need to install
the `ExternalArtifact` CRD separately, as it will be available out of the box with Flux.

The `ExternalArtifact` API specifications will be published to the Flux documentation website,
under the `source.toolkit.fluxcd.io` API reference section.

The Flux `Kustomization` and `HelmRelease` APIs will be extended to support the `ExternalArtifact` kind
as a valid `sourceRef.kind` and `chartRef.kind`. The `kustomize-controller` and `helm-controller`
will gain the ability to consume artifacts from `ExternalArtifact` and watch for revision changes.

The `flux trace` command will be extended to support the `ExternalArtifact` API, allowing Flux users
to trace any Kubernetes resource in-cluster that originates from an `ExternalArtifact` and see the
`sourceRef` information that points to the original source.

The `flux` CLI will implement the `flux get externalartifact` command for listing and status checking
of `ExternalArtifact` custom resources in the cluster.

### Feature Gate

While the `ExternalArtifact` API will be available out of the box with Flux,
the ability for `kustomize-controller` and `helm-controller` to consume artifacts
from `ExternalArtifact` resources will be behind a feature gate called `ExternalArtifact`.

The feature gate will be disabled by default and can be enabled by setting
the `--feature-gates=ExternalArtifact=true` flag on the `kustomize-controller`
and `helm-controller` deployments. This allows cluster administrators to
control the adoption of the `ExternalArtifact` feature in their clusters.

## Implementation History

<!--
Major milestones in the lifecycle of the RFC such as:
- The first Flux release where an initial version of the RFC was available.
- The version of Flux where the RFC graduated to general availability.
- The version of Flux where the RFC was retired or superseded.
-->