mirror of https://github.com/fluxcd/flux2.git
You cannot select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
198 lines
7.2 KiB
Markdown
198 lines
7.2 KiB
Markdown
3 years ago
|
# RFC-0003 Flux Multi-Tenancy Mode
|
||
|
|
||
|
## Summary
|
||
|
|
||
|
For multi-tenant environments, we want to offer an easy way of configuring Flux to enforce tenant isolation
|
||
|
(as defined by the Soft Multi-Tenancy model from RFC-0001).
|
||
|
|
||
|
When running in the multi-tenant mode, Flux will lock down access to sources (as defined by RFC-0002),
|
||
|
and will use the tenant service account instead of defaulting to `cluster-admin`.
|
||
|
|
||
|
From an end-user perspective, the multi-tenancy mode means that:
|
||
|
|
||
|
- Platform admins have to create a Kubernetes service account and RBAC in each namespace where
|
||
|
Flux performs source-to-cluster reconciliation on behalf of tenants.
|
||
|
By default, Flux will have no permissions to reconcile the tenants sources onto clusters.
|
||
|
- Source owners have to specify with which tenants they wish to share their sources.
|
||
|
By default, nothing is shared between tenants.
|
||
|
|
||
|
## Motivation
|
||
|
|
||
|
As of Flux v0.23.0, configuring Flux for soft multi-tenancy requires additional tooling such as Kyverno or OPA Gatekeeper
|
||
|
to overcome caveats such as:
|
||
|
- Flux does not require for a service account name to be specified on Flux custom resources that perform
|
||
|
source-to-cluster reconciliation. When a service account is not specified, Flux defaults to cluster-admin.
|
||
|
- Flux does not prevent tenants from accessing known sources outside of their namespaces.
|
||
|
- Flux does not prevent tenants from subscribing to other tenant's events.
|
||
|
|
||
|
Flux users have been asking for a way to enforce multi-tenancy
|
||
|
without having to use 3rd party validation webhooks e.g.
|
||
|
[fluxcd/kustomize-controller#422](https://github.com/fluxcd/kustomize-controller/issues/422).
|
||
|
|
||
|
### Goals
|
||
|
|
||
|
- Enforce service account impersonation for source-to-cluster reconciliation.
|
||
|
- Enforce ACLs for cross-namespace access to sources.
|
||
|
|
||
|
### Non-Goals
|
||
|
|
||
|
- Enforce tenant's workload isolation with network policies and pod security standards as described
|
||
|
[here](https://kubernetes.io/blog/2021/04/15/three-tenancy-models-for-kubernetes/#security-considerations).
|
||
|
|
||
|
## Proposal
|
||
|
|
||
|
### User Stories
|
||
|
|
||
|
#### Story 1
|
||
|
|
||
|
> As a platform admin, I want to install Flux with lowest privilege/permission level possible.
|
||
|
|
||
|
#### Story 2
|
||
|
|
||
|
> As a platform admin, I want to give tenants full control over their assigned namespaces.
|
||
|
> So that tenants could use their own repositories and manager the app delivery with Flux.
|
||
|
|
||
|
#### Story 3
|
||
|
|
||
|
> As a platform admin, I want to prevent tenants from changing the cluster-wide configuration.
|
||
|
> If a tenant adds to their repository a cluster-scoped resource such as a namespace or cluster role,
|
||
|
> Flux should reject the change and notify the tenant that this operation is not allowed.
|
||
|
|
||
|
### Multi-tenant Bootstrap
|
||
|
|
||
|
When bootstrapping Flux, platform admins should have the option to lock down Flux for multi-tenant environments e.g.:
|
||
|
|
||
|
```shell
|
||
|
flux bootstrap --security-profile=multi-tenant
|
||
|
```
|
||
|
|
||
|
The security profile flag accepts two values: `single-tenant` and `multi-tenant`.
|
||
|
Platform admins may switch between the two modes at any time, either by rerunning bootstrap
|
||
|
or by patching the Flux manifests in Git.
|
||
|
|
||
|
The `multi-tenant` profile is just a shortcut to setting the following container args in the Flux deployment manifests:
|
||
|
|
||
|
```yaml
|
||
|
containers:
|
||
|
- name: manager
|
||
|
args:
|
||
|
- --default-service-account=flux
|
||
|
- --enable-source-acl=true
|
||
|
```
|
||
|
|
||
|
And for disabling cross-namespace references when using the notification API:
|
||
|
|
||
|
```yaml
|
||
|
kind: Deployment
|
||
|
metadata:
|
||
|
name: notification-controller
|
||
|
spec:
|
||
|
template:
|
||
|
spec:
|
||
|
containers:
|
||
|
- name: manager
|
||
|
args:
|
||
|
- --same-namespace-refs=true
|
||
|
```
|
||
|
|
||
|
When running in the `multi-tenant` mode, Flux behaves differently:
|
||
|
|
||
|
- The source-to-cluster reconciliation no longer runs under the service account of
|
||
|
the Flux controllers. The controller service account, is only used to impersonate
|
||
|
the service account specified in the Flux custom resources (`Kustomizations`, `HelmReleases`).
|
||
|
- When no service account name is specified in a Flux custom resource,
|
||
|
a default will be used e.g. `system:serviceaccount:<tenant-namespace>:flux`.
|
||
|
- When a Flux custom resource (`Kustomizations`, `HelmReleases`, `ImagePolicies`, `ImageUpdateAutomations`)
|
||
|
refers to a source in a different namespace, access is granted based the source access control list.
|
||
|
If no ACL is defined for a source, cross-namespace access is denied.
|
||
|
- When a Flux notification (`Alerts`, `Receivers`)
|
||
|
refers to a resource in a different namespace, access is denied.
|
||
|
|
||
|
### Tenants Onboarding
|
||
|
|
||
|
When onboarding tenants, platform admins should have the option to assign namespaces, set
|
||
|
permissions and register the tenants repositories onto clusters in a declarative manner.
|
||
|
|
||
|
The Flux CLI offers an easy way of generating all the Kubernetes manifests needed to onboard tenants:
|
||
|
|
||
|
- `flux create tenant` command generates namespaces, service accounts and Kubernetes RBAC
|
||
|
with restricted access to the cluster resources, given tenants access only to their namespaces.
|
||
|
- `flux create secret git` command generates SSH keys used by Flux to clone the tenants repositories.
|
||
|
- `flux create source git` command generates the configuration that tells Flux which repositories belong to tenants.
|
||
|
- `flux create kustomization` command generates the configuration that tells Flux how to reconcile the manifests found in the tenants repositories.
|
||
|
|
||
|
All the above commands have an `--export` flag for generating the Kubernetes resources in YAML format.
|
||
|
The platform admins should place the generated manifests in the repository that defines the cluster(s) desired state.
|
||
|
|
||
|
Here is an example of the generated manifests:
|
||
|
|
||
|
```yaml
|
||
|
---
|
||
|
apiVersion: v1
|
||
|
kind: Namespace
|
||
|
metadata:
|
||
|
name: tenant1
|
||
|
---
|
||
|
apiVersion: v1
|
||
|
kind: ServiceAccount
|
||
|
metadata:
|
||
|
name: flux
|
||
|
namespace: tenant1
|
||
|
---
|
||
|
apiVersion: rbac.authorization.k8s.io/v1
|
||
|
kind: RoleBinding
|
||
|
metadata:
|
||
|
name: flux
|
||
|
namespace: tenant1
|
||
|
roleRef:
|
||
|
apiGroup: rbac.authorization.k8s.io
|
||
|
kind: ClusterRole
|
||
|
name: cluster-admin
|
||
|
subjects:
|
||
|
- kind: ServiceAccount
|
||
|
name: flux
|
||
|
namespace: tenant1
|
||
|
---
|
||
|
apiVersion: source.toolkit.fluxcd.io/v1beta1
|
||
|
kind: GitRepository
|
||
|
metadata:
|
||
|
name: tenant1
|
||
|
namespace: tenant1
|
||
|
spec:
|
||
|
interval: 5m0s
|
||
|
ref:
|
||
|
branch: main
|
||
|
secretRef:
|
||
|
name: tenant1-git-auth
|
||
|
url: ssh://git@github.com/org/tenant1
|
||
|
---
|
||
|
apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
|
||
|
kind: Kustomization
|
||
|
metadata:
|
||
|
name: tenant1
|
||
|
namespace: tenant1
|
||
|
spec:
|
||
|
interval: 10m0s
|
||
|
path: ./
|
||
|
prune: true
|
||
|
serviceAccountName: flux
|
||
|
sourceRef:
|
||
|
kind: GitRepository
|
||
|
name: tenant1
|
||
|
```
|
||
|
|
||
|
Note that the [cluster-admin](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#user-facing-roles)
|
||
|
role is used in a `RoleBinding`, this only gives full control over every resource in the role binding's namespace.
|
||
|
|
||
|
Once the tenants repositories are registered on the cluster(s), the tenants can configure their app delivery
|
||
|
in Git using Kubernetes namespace-scoped resources such as `Deployments`, `Services`, Flagger `Canaries`,
|
||
|
Flux `Kustomizations`, `HelmReleases`, `ImageUpdateAutomations`, `Alerts`, `Receivers`, etc.
|
||
|
|
||
|
## Alternatives
|
||
|
|
||
|
Instead of introducing the security profile flag to `flux bootstrap`,
|
||
|
we could document how to patch each controller deployment with Kustomize.
|
||
|
|
||
|
Having an easy way of locking down Flux with a single flag, make users aware of the security implications
|
||
|
and improves the user experience.
|