mirror of https://github.com/fluxcd/flux2.git
Define Flux tenancy models
Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>pull/2086/head
Stefan Prodan
3 years ago
parent
e07558f5b7
commit
d23d87ac94
@ -0,0 +1,242 @@
|
||||
# RFC-0001 Flux Multi-Tenancy
|
||||
|
||||
## Summary
|
||||
|
||||
The main goal of this RFC is to define the Kubernetes tenancy models supported by Flux.
|
||||
|
||||
## Motivation
|
||||
|
||||
As of Flux v0.23.0, the documentation contains references to account impersonation, reconciliation isolation and other
|
||||
multi-tenancy related features without clearly defining which tenancy models are supported by Flux and how
|
||||
they relate to source access control and Kubernetes role-based access control.
|
||||
|
||||
### Goals
|
||||
|
||||
- Define multi-tenancy from a Flux user perspective.
|
||||
- List the types of users that interact with Flux in regard to multi-tenancy.
|
||||
- List the tenancy models supported by Flux.
|
||||
- Explain the differences between tenancy models.
|
||||
|
||||
### Non-Goals
|
||||
|
||||
- Explain in detail the Kubernetes architecture in regard to multi-tenancy.
|
||||
- Provide an end-to-end workflow of how to setup multi-tenancy with Flux.
|
||||
|
||||
## Introduction
|
||||
|
||||
Flux allows different organizations and/or teams to share the same Kubernetes control plane to deliver applications
|
||||
in a GitOps manner. Flux enables segmentation and isolation of resources across tenants by leveraging
|
||||
Kubernetes Cluster API, namespaces and role-based access control.
|
||||
|
||||
## User Roles
|
||||
|
||||
There are two types of users that interact with Flux: platform admins and tenants.
|
||||
Besides installing Flux, all the other operations (deploy applications, configure ingress, policies, etc)
|
||||
do not require users to have direct access to the Kubernetes API. Flux acts as a proxy between users and
|
||||
the Kubernetes API, using Git as source of truth for the cluster desired state. Changes to the clusters
|
||||
and workloads configuration can be made in a collaborative manner, where the various teams responsible for
|
||||
the delivery process propose, review and approve changes via pull request workflows.
|
||||
|
||||
### Platform Admins
|
||||
|
||||
The platform admins have unrestricted access to Kubernetes API.
|
||||
They are responsible for installing Flux and granting Flux
|
||||
access to the sources (Git, Helm, OCI repositories) that make up the cluster(s) control plane desired state.
|
||||
The repository(s) owned by the platform admins are reconciled on the cluster(s) by Flux, under
|
||||
the [cluster-admin](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#user-facing-roles)
|
||||
Kubernetes cluster role.
|
||||
|
||||
Example of operations performed by cluster admins:
|
||||
|
||||
- Bootstrap Flux onto cluster(s).
|
||||
- Extend the Kubernetes API with custom resource definitions and validation webhooks.
|
||||
- Configure various controllers for ingress, storage, logging, monitoring, progressive delivery, etc.
|
||||
- Set up namespaces for tenants and define their level of access with Kubernetes RBAC.
|
||||
- Onboard tenants by registering their Git repositories with Flux.
|
||||
|
||||
### Tenants
|
||||
|
||||
The tenants have restricted access to the cluster(s) according to the Kubernetes RBAC configured
|
||||
by the platform admins. The repositories owned by tenants are reconciled on the cluster(s) by Flux,
|
||||
under the Kubernetes account(s) assigned by platform admins.
|
||||
|
||||
Example of operations performed by tenants:
|
||||
|
||||
- Register their sources with Flux (`GitRepositories`, `HelmRepositories` and `Buckets`).
|
||||
- Deploy workload(s) into their namespace(s) using Flux custom resources (`Kustomizations` and `HelmReleases`).
|
||||
- Automate application updates using Flux custom resources (`ImageRepositories`, `ImagePolicies` and `ImageUpdateAutomations`).
|
||||
- Configure the release pipeline(s) using Flagger custom resources (`Canaries` and `MetricsTemplates`).
|
||||
- Setup webhooks and alerting for their release pipeline(s) using Flux custom resources (`Receivers` and `Alerts`).
|
||||
|
||||
## Tenancy Models
|
||||
|
||||
The Kubernetes tenancy models supported by Flux are: soft multi-tenancy and hard multi-tenancy.
|
||||
|
||||
For an overview of the Kubernetes multi-tenant architecture please consult the following documentation:
|
||||
|
||||
- [Three Tenancy Models For Kubernetes](https://kubernetes.io/blog/2021/04/15/three-tenancy-models-for-kubernetes/)
|
||||
- [GKE multi-tenancy overview](https://cloud.google.com/kubernetes-engine/docs/concepts/multitenancy-overview)
|
||||
- [EKS multi-tenancy best practices](https://aws.github.io/aws-eks-best-practices/security/docs/multitenancy/)
|
||||
|
||||
### Soft Multi-Tenancy
|
||||
|
||||
With soft multi-tenancy, the platform admins use Kubernetes constructs such as namespaces, accounts,
|
||||
roles and role bindings to create a logical separation between tenants.
|
||||
|
||||
When Flux deploys workloads from a repository belonging to a tenant, it uses the Kubernetes account assigned to that
|
||||
tenant to perform the git-to-cluster reconciliation. By leveraging Kubernetes RBAC, Flux can ensure
|
||||
that the operations performed by tenants are restricted to their namespaces.
|
||||
|
||||
Note that with this model, tenants share cluster-wide resources such as
|
||||
`ClusterRoles`, `CustomResourceDefinitions`, `IngressClasses`, `StorageClasses`,
|
||||
and they cannot create or alter these resources.
|
||||
If a tenant adds a cluster-scoped resource definition to their repository,
|
||||
Flux will fail the git-to-cluster reconciliation due to Kubernetes RBAC restrictions.
|
||||
|
||||
To restrict the reconciliation of tenant's sources, a Kubernetes service account name can be specified
|
||||
in Flux `Kustomizations` and `HelmReleases` under `.spec.serviceAccountName`. Please consult the Flux
|
||||
documentation for more details:
|
||||
|
||||
- [Kustomization API: Role-based access control](https://fluxcd.io/docs/components/kustomize/kustomization/#role-based-access-control)
|
||||
- [HelmRelease API: Role-based access control](https://fluxcd.io/docs/components/helm/helmreleases/#role-based-access-control)
|
||||
- [Flux multi-tenancy example repository](https://github.com/fluxcd/flux2-multi-tenancy)
|
||||
|
||||
Note that with soft multi-tenancy, true tenant isolation requires security measures beyond Kubernetes RBAC.
|
||||
Please refer to the Kubernetes [security considerations documentation](https://kubernetes.io/blog/2021/04/15/three-tenancy-models-for-kubernetes/#security-considerations)
|
||||
for more details on how to harden shared clusters.
|
||||
|
||||
#### Tenants Onboarding
|
||||
|
||||
When onboarding tenants, platform admins have the option to assign namespaces, set
|
||||
permissions and register the tenants main repositories onto clusters in a declarative manner.
|
||||
|
||||
The Flux CLI offers an easy way of generating all the Kubernetes manifests needed to onboard tenants:
|
||||
|
||||
- `flux create tenant` command generates namespaces, service accounts and Kubernetes RBAC
|
||||
with restricted access to the cluster resources, given tenants access only to their namespaces.
|
||||
- `flux create secret git` command generates SSH keys used by Flux to clone the tenants repositories.
|
||||
- `flux create source git` command generates the configuration that tells Flux which repositories belong to tenants.
|
||||
- `flux create kustomization` command generates the configuration that tells Flux how to reconcile the manifests found in the tenants repositories.
|
||||
|
||||
All the above commands have an `--export` flag for generating the Kubernetes resources in YAML format.
|
||||
The platform admins should place the generated manifests in the repository that defines the cluster(s) desired state.
|
||||
|
||||
Here is an example of the generated manifests:
|
||||
|
||||
```yaml
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
metadata:
|
||||
name: tenant1
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: ServiceAccount
|
||||
metadata:
|
||||
name: flux
|
||||
namespace: tenant1
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: RoleBinding
|
||||
metadata:
|
||||
name: flux
|
||||
namespace: tenant1
|
||||
roleRef:
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
kind: ClusterRole
|
||||
name: cluster-admin
|
||||
subjects:
|
||||
- kind: ServiceAccount
|
||||
name: flux
|
||||
namespace: tenant1
|
||||
---
|
||||
apiVersion: source.toolkit.fluxcd.io/v1beta1
|
||||
kind: GitRepository
|
||||
metadata:
|
||||
name: tenant1
|
||||
namespace: tenant1
|
||||
spec:
|
||||
interval: 5m0s
|
||||
ref:
|
||||
branch: main
|
||||
secretRef:
|
||||
name: tenant1-git-auth
|
||||
url: ssh://git@github.com/org/tenant1
|
||||
---
|
||||
apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
name: tenant1
|
||||
namespace: tenant1
|
||||
spec:
|
||||
interval: 10m0s
|
||||
path: ./
|
||||
prune: true
|
||||
serviceAccountName: flux
|
||||
sourceRef:
|
||||
kind: GitRepository
|
||||
name: tenant1
|
||||
```
|
||||
|
||||
Note that the [cluster-admin](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#user-facing-roles)
|
||||
role is used in a `RoleBinding`, this only gives full control over every resource in the role binding's namespace.
|
||||
|
||||
Once the tenants main repositories are registered on the cluster(s), the tenants can configure their app delivery
|
||||
in Git using Kubernetes namespace-scoped resources such as `Deployments`, `Services`, Flagger `Canaries`,
|
||||
Flux `GitRepositories`, `Kustomizations`, `HelmRepositories`, `HelmReleases`, `ImageUpdateAutomations`,
|
||||
`Alerts`, `Receivers`, etc.
|
||||
|
||||
#### Caveats
|
||||
|
||||
As of v0.23.0, Flux does not enforce a service account to be specified on Flux `Kustomizations` and `HelmReleases`.
|
||||
When a service account is not specified, Flux defaults to cluster-admin.
|
||||
In order to enforce the tenant isolation, an admission controller such as Kyverno or OPA Gatekeeper must be used
|
||||
to make the `.spec.serviceAccountName` a required field for the Flux custom resources created by tenants.
|
||||
|
||||
We provide an [example](https://github.com/fluxcd/flux2-multi-tenancy/blob/main/infrastructure/kyverno-policies/flux-multi-tenancy.yaml)
|
||||
for enforcing service accounts using a Kyverno cluster policy.
|
||||
|
||||
As of v0.23.0, Flux allows for `Kustomizations` and `HelmReleases` to reference sources
|
||||
(`GitRepositories`, `HelmRepositories` and `Buckets`) across namespaces.
|
||||
In order to prevent tenants from accessing each other sources, an admission controller such as Kyverno or OPA Gatekeeper
|
||||
must be used to block cross-namespace references.
|
||||
|
||||
We provide an [example](https://github.com/fluxcd/flux2-multi-tenancy/blob/main/infrastructure/kyverno-policies/flux-multi-tenancy.yaml)
|
||||
for blocking source cross-namespace references using a Kyverno cluster policy.
|
||||
|
||||
### Hard Multi-Tenancy
|
||||
|
||||
With hard multi-tenancy, the platform admins use Kubernetes Cluster API to create dedicated clusters for each tenant.
|
||||
The Flux instance installed on the management cluster is responsible
|
||||
for reconciling the cluster definitions belonging to tenants.
|
||||
|
||||
To enable GitOps for the tenant's clusters, the platform admins can configure the Flux instance running on the
|
||||
management cluster to connect to the tenant's cluster using the `kubeConfig` generated by the Cluster API provider.
|
||||
|
||||
To configure Flux reconciliation of remote clusters, a Kubernetes secret containing a `kubeConfig` can be specified
|
||||
in Flux `Kustomizations` and `HelmReleases` under `.spec.kubeConfig.secretRef`. Please consult the Flux API
|
||||
documentation for more details:
|
||||
|
||||
- [Kustomization API: Remote Clusters](https://fluxcd.io/docs/components/kustomize/kustomization/#remote-clusters--cluster-api)
|
||||
- [HelmRelease API: Remote Clusters](https://fluxcd.io/docs/components/helm/helmreleases/#remote-clusters--cluster-api)
|
||||
|
||||
Note that with hard multi-tenancy, tenants have full access to cluster-wide resources, so they have the option
|
||||
to manage Flux independently of platform admins, by deploying a Flux instance on each cluster.
|
||||
|
||||
#### Caveats
|
||||
|
||||
When using a Kubernetes Cluster API provider, the `kubeConfig` secret is automatically generated and Flux can
|
||||
make use of it without any manual actions. For clusters created by other means than Cluster API, the
|
||||
platform team has to create the `kubeConfig` secrets to allow Flux access to the remote clusters.
|
||||
|
||||
As of Flux v0.23.0, we don't provide any guidance for cluster admins on how to generate the `kubeConfig` secrets.
|
||||
|
||||
## Implementation History
|
||||
|
||||
- Soft multi-tenancy based on service account impersonation was first released in flux2 **v0.0.1**.
|
||||
- Generating namespaces and RBAC for defining tenants with `flux create tenant` was first released in flux2 **v0.1.0**.
|
||||
- Hard multi-tenancy based on remote cluster reconciliation was first released in flux2 **v0.2.0**.
|
||||
- Soft multi-tenancy end-to-end workflow example was first published on 27 Nov 2020 at
|
||||
[fluxcd/flux2-multi-tenancy](https://github.com/fluxcd/flux2-multi-tenancy).
|
||||
- Soft multi-tenancy [CVE-2021-41254](https://github.com/fluxcd/kustomize-controller/security/advisories/GHSA-35rf-v2jv-gfg7)
|
||||
"Privilege escalation to cluster admin on multi-tenant environments" was fixed in flux2 **v0.15.0**.
|
||||
Loading…
Reference in New Issue