From e0bc754ad04cf2783359633aa2416fae0a7cdfd5 Mon Sep 17 00:00:00 2001 From: Stefan Prodan Date: Fri, 17 Dec 2021 11:43:58 +0200 Subject: [PATCH] Refer to authorisation model in RFC-0001 Signed-off-by: Stefan Prodan --- .../README.md | 140 +++++------------- 1 file changed, 37 insertions(+), 103 deletions(-) rename rfcs/{0001-multi-tenancy => 0004-multi-tenancy}/README.md (72%) diff --git a/rfcs/0001-multi-tenancy/README.md b/rfcs/0004-multi-tenancy/README.md similarity index 72% rename from rfcs/0001-multi-tenancy/README.md rename to rfcs/0004-multi-tenancy/README.md index 45b597f0..7af416ce 100644 --- a/rfcs/0001-multi-tenancy/README.md +++ b/rfcs/0004-multi-tenancy/README.md @@ -1,4 +1,10 @@ -# RFC-0001 Flux Multi-Tenancy +# RFC-0004 Flux Multi-Tenancy + +**Status:** provisional + +**Creation date:** 2021-11-15 + +**Last update:** 2021-12-17 ## Summary @@ -7,10 +13,10 @@ models for multi-tenancy, and gives reference implementations for those models. ## Motivation -To this point, the Flux project has provided [examples of -multi-tenancy](https://github.com/fluxcd/flux2-multi-tenancy/tree/v0.1.0), but not explained exactly +To this point, the Flux project has provided [examples of multi-tenancy][mt], but not explained exactly how they relate to Flux's authorisation model. This RFC explains two multi-tenancy implementations, -their security properties, and how they are implemented within the authorisation model. +their security properties, and how they are implemented within the authorisation model +as defined in [RFC-0001](https://github.com/fluxcd/flux2/pull/2212). ### Goals @@ -22,7 +28,7 @@ their security properties, and how they are implemented within the authorisation ### Non-Goals - Give an exhaustive account of multi-tenancy implementations in general. -- Provide an [end-to-end workflow](](https://github.com/fluxcd/flux2-multi-tenancy/tree/v0.1.0)) of +- Provide an [end-to-end workflow][mt] of how to set up multi-tenancy with Flux. ## Introduction @@ -54,14 +60,18 @@ to a cluster. These controllers are subject to authorisation on two counts: - when accessing Kubernetes resources that are needed for a particular "apply" operation -- for example, a secret referenced in the field `.spec.valuesFrom` in a `HelmRelease`; - - when creating, updating and deleting Kubernetes resources in the process of applying a piece of - configuration. + - when creating, watching, updating and deleting Kubernetes resources + in the process of applying a piece of configuration. To give users control over this authorisation, these two controllers will _impersonate_ (assume the -identity of) a service account mentioned in the apply specification (e.g., [the field -`.spec.serviceAccountName` in a `Kustomization` object][serviceAccountName]) for both accessing -resources and applying configuration. This lets a user constrain the operations mentioned above with -RBAC. +identity of) a service account mentioned in the apply specification (e.g., the field +`.spec.serviceAccountName` in a [`Kustomization` object][kcsa] +or in a [`HelmRelease` object][hcsa]) for both accessing resources and applying configuration. +This lets a user constrain the operations mentioned above with RBAC. + +As stated in [RFC-0003](https://github.com/fluxcd/flux2/pull/2093), +the platform admins can configure Flux to enforce service account impersonation +by setting a default service account name when `.spec.serviceAccountName` is not specified. #### Remote apply @@ -72,9 +82,6 @@ the client used to apply the specified set of configuration. The effect of this configuration will be applied as the user given in the kubeconfig; often this is a user with the `cluster-admin` role bound to it, but not necessarily so. -[serviceAccountName]: https://fluxcd.io/docs/components/kustomize/api/#kustomize.toolkit.fluxcd.io/v1beta2.KustomizationSpec -[kubeconfig]: https://fluxcd.io/docs/components/kustomize/api/#kustomize.toolkit.fluxcd.io/v1beta2.KubeConfig - ## Assumptions made by the multi-tenancy models ### User Roles @@ -146,9 +153,9 @@ To restrict the reconciliation of tenant's sources, a Kubernetes service account in Flux `Kustomizations` and `HelmReleases` under `.spec.serviceAccountName`. Please consult the Flux documentation for more details: -- [Kustomization API: Role-based access control](https://fluxcd.io/docs/components/kustomize/kustomization/#role-based-access-control) -- [HelmRelease API: Role-based access control](https://fluxcd.io/docs/components/helm/helmreleases/#role-based-access-control) -- [Flux multi-tenancy example repository](https://github.com/fluxcd/flux2-multi-tenancy) +- [Kustomization API: Role-based access control][kcsa] +- [HelmRelease API: Role-based access control][hcsa] +- [Flux multi-tenancy example repository][mt] Note that with soft multi-tenancy, true tenant isolation requires security measures beyond Kubernetes RBAC. Please refer to the Kubernetes [security considerations documentation](https://kubernetes.io/blog/2021/04/15/three-tenancy-models-for-kubernetes/#security-considerations) @@ -167,100 +174,22 @@ The Flux CLI offers an easy way of generating all the Kubernetes manifests neede - `flux create source git` command generates the configuration that tells Flux which repositories belong to tenants. - `flux create kustomization` command generates the configuration that tells Flux how to reconcile the manifests found in the tenants repositories. -All the above commands have an `--export` flag for generating the Kubernetes resources in YAML format. -The platform admins should place the generated manifests in the repository that defines the cluster(s) desired state. - -Here is an example of the generated manifests: - -```yaml ---- -apiVersion: v1 -kind: Namespace -metadata: - name: tenant1 ---- -apiVersion: v1 -kind: ServiceAccount -metadata: - name: flux - namespace: tenant1 ---- -apiVersion: rbac.authorization.k8s.io/v1 -kind: RoleBinding -metadata: - name: flux - namespace: tenant1 -roleRef: - apiGroup: rbac.authorization.k8s.io - kind: ClusterRole - name: cluster-admin -subjects: - - kind: ServiceAccount - name: flux - namespace: tenant1 ---- -apiVersion: source.toolkit.fluxcd.io/v1beta1 -kind: GitRepository -metadata: - name: tenant1 - namespace: tenant1 -spec: - interval: 5m0s - ref: - branch: main - secretRef: - name: tenant1-git-auth - url: ssh://git@github.com/org/tenant1 ---- -apiVersion: kustomize.toolkit.fluxcd.io/v1beta2 -kind: Kustomization -metadata: - name: tenant1 - namespace: tenant1 -spec: - interval: 10m0s - path: ./ - prune: true - serviceAccountName: flux - sourceRef: - kind: GitRepository - name: tenant1 -``` - -Note that the [cluster-admin](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#user-facing-roles) -role is used in a `RoleBinding`; this only gives full control over every resource in the role binding's namespace. - Once the tenants main repositories are registered on the cluster(s), the tenants can configure their app delivery in Git using Kubernetes namespace-scoped resources such as `Deployments`, `Services`, Flagger `Canaries`, Flux `GitRepositories`, `Kustomizations`, `HelmRepositories`, `HelmReleases`, `ImageUpdateAutomations`, `Alerts`, `Receivers`, etc. -#### Caveats - -As of v0.23.0, Flux does not enforce a service account to be specified on Flux `Kustomizations` and `HelmReleases`. -When a service account is not specified, Flux defaults to cluster-admin. -In order to enforce the tenant isolation, an admission controller such as Kyverno or OPA Gatekeeper must be used -to make the `.spec.serviceAccountName` a required field for the Flux custom resources created by tenants. - -We provide an [example](https://github.com/fluxcd/flux2-multi-tenancy/blob/main/infrastructure/kyverno-policies/flux-multi-tenancy.yaml) -for enforcing service accounts using a Kyverno cluster policy. - -As of v0.23.0, Flux allows for `Kustomizations` and `HelmReleases` to reference sources -(`GitRepositories`, `HelmRepositories` and `Buckets`) across namespaces. -In order to prevent tenants from accessing each other sources, an admission controller such as Kyverno or OPA Gatekeeper -must be used to block cross-namespace references. - -We provide an [example](https://github.com/fluxcd/flux2-multi-tenancy/blob/main/infrastructure/kyverno-policies/flux-multi-tenancy.yaml) -for blocking source cross-namespace references using a Kyverno cluster policy. - ### Hard Multi-Tenancy -With hard multi-tenancy, the platform admins use Kubernetes Cluster API to create dedicated clusters for each tenant. -The Flux instance installed on the management cluster is responsible -for reconciling the cluster definitions belonging to tenants. +With hard multi-tenancy, the platform admins create dedicated clusters for each tenant. + +When the tenants's clusters are created with Kubernetes Cluster API, the Flux instance +installed on the management cluster is responsible for reconciling the cluster +definitions belonging to tenants. To enable GitOps for the tenant's clusters, the platform admins can configure the Flux instance running on the -management cluster to connect to the tenant's cluster using the `kubeConfig` generated by the Cluster API provider. +management cluster to connect to the tenant's cluster using the kubeconfig generated by the Cluster API provider +or by creating kubeconfig secrets for the clusters created by other means than Cluster API. To configure Flux reconciliation of remote clusters, a Kubernetes secret containing a `kubeConfig` can be specified in Flux `Kustomizations` and `HelmReleases` under `.spec.kubeConfig.secretRef`. Please consult the Flux API @@ -278,7 +207,7 @@ When using a Kubernetes Cluster API provider, the `kubeConfig` secret is automat make use of it without any manual actions. For clusters created by other means than Cluster API, the platform team has to create the `kubeConfig` secrets to allow Flux access to the remote clusters. -As of Flux v0.23.0, we don't provide any guidance for cluster admins on how to generate the `kubeConfig` secrets. +As of Flux v0.24 (Nov 2021), we don't provide any guidance for cluster admins on how to generate the `kubeConfig` secrets. ## Implementation History @@ -289,3 +218,8 @@ As of Flux v0.23.0, we don't provide any guidance for cluster admins on how to g [fluxcd/flux2-multi-tenancy](https://github.com/fluxcd/flux2-multi-tenancy). - Soft multi-tenancy [CVE-2021-41254](https://github.com/fluxcd/kustomize-controller/security/advisories/GHSA-35rf-v2jv-gfg7) "Privilege escalation to cluster admin on multi-tenant environments" was fixed in flux2 **v0.15.0**. + +[mt]: https://github.com/fluxcd/flux2-multi-tenancy/tree/v0.1.0 +[kcsa]: https://fluxcd.io/docs/components/kustomize/kustomization/#role-based-access-control +[hcsa]: https://fluxcd.io/docs/components/helm/helmreleases/#role-based-access-control +[kubeconfig]: https://fluxcd.io/docs/components/kustomize/api/#kustomize.toolkit.fluxcd.io/v1beta2.KubeConfig