mirror of https://github.com/fluxcd/flux2.git
Merge 763f262590 into e09ba7a8e6
Paulo Gomes
3 months ago
committed by
GitHub
commit
e72d5c7bd6
@ -0,0 +1,344 @@
|
|||||||
|
# RFC-NNNN Trust Store for TLS and SSH
|
||||||
|
|
||||||
|
**Status:** provisional
|
||||||
|
|
||||||
|
**Creation date:** 2022-12-02
|
||||||
|
|
||||||
|
**Last update:** 2022-12-02
|
||||||
|
|
||||||
|
## Summary
|
||||||
|
|
||||||
|
Consolidates and formalizes the supported ways to establish trusted connections
|
||||||
|
with remote servers via Transport Layer Security (TLS) and Secure Shell (SSH).
|
||||||
|
Resulting on new ways to configure trust, and allowing administrators the
|
||||||
|
capability to disable some of the existing options.
|
||||||
|
|
||||||
|
## Motivation
|
||||||
|
|
||||||
|
The current model could be improved by allowing for controller-level trust
|
||||||
|
configurations, so that multiple objects connecting to the same server don't
|
||||||
|
need to specify overrides. The approach aligns with both TLS and SSH canonical
|
||||||
|
OS level implementations, in which they rely on a global trust store to define
|
||||||
|
machine level trust, but users and applications can further expand on trusted
|
||||||
|
servers (or CAs), when not blocked by administrators.
|
||||||
|
|
||||||
|
Known Hosts (used by SSH connections), and CA Bundles (for TLS), are not
|
||||||
|
particularly sensitive information - when leaving aside privacy considerations.
|
||||||
|
Before this RFC, the officially supported approach leans on secrets to pass this
|
||||||
|
information to the controllers. The same secret is also use to provide user
|
||||||
|
credentials, which is more sensitive in nature, making this sub-optimal from a
|
||||||
|
security stand-point.
|
||||||
|
|
||||||
|
### Goals
|
||||||
|
|
||||||
|
- Consolidate the officially supported trust settings across the Flux ecosystem.
|
||||||
|
- Formalize support for configuring trust at controller-level.
|
||||||
|
- Add toggle to block object-level trust overrides.
|
||||||
|
- Enable users to surface trust information securely.
|
||||||
|
|
||||||
|
### Non-Goals
|
||||||
|
|
||||||
|
- Maintain backwards compatibility with older versions of Flux.
|
||||||
|
|
||||||
|
## Proposal
|
||||||
|
|
||||||
|
For configuring system-wide trust, Flux would rely on the well-established OS-level
|
||||||
|
trust stores. When dynamically mounting of the trust store is required, it will be
|
||||||
|
enabled by using Kubernetes `Secret` and `ConfigMap` mounting. When immutable trust
|
||||||
|
store is required, users can build their own version of the controllers, with their
|
||||||
|
baked-in settings.
|
||||||
|
|
||||||
|
TLS and SSH use different techniques to establish the identity of remote servers,
|
||||||
|
each relying on its own trust store.
|
||||||
|
|
||||||
|
The sections below will dive into the specifics of each one, highlighting their
|
||||||
|
details, changes required and example of the proposed usage.
|
||||||
|
|
||||||
|
A new way to configure object-level Trust Store overrides is also being proposed,
|
||||||
|
in combination with a controller level toggle to disable it.
|
||||||
|
|
||||||
|
### SSH
|
||||||
|
|
||||||
|
In SSH, the remote server identity is based on [Trust on first use]. At first
|
||||||
|
connection to a new server, the user confirms whether or not to trust that server
|
||||||
|
based on the server's Public key fingerprint.
|
||||||
|
|
||||||
|
In the context of Flux, which provides no user interaction, if the remote server
|
||||||
|
finger print is not configured within the provided set of Known Hosts, the
|
||||||
|
connection is aborted.
|
||||||
|
|
||||||
|
#### Controller-level Known Hosts
|
||||||
|
|
||||||
|
For setting controller level Known Hosts, we propose the use of the existing
|
||||||
|
Linux file in disk: [/etc/ssh/ssh_known_hosts].
|
||||||
|
|
||||||
|
Users would be able to configure the OS level trust store by mounting either
|
||||||
|
a `ConfigMap` or `Secret` directly into the Flux Controllers.
|
||||||
|
|
||||||
|
`ConfigMap` example:
|
||||||
|
```yaml
|
||||||
|
apiVersion: v1
|
||||||
|
kind: ConfigMap
|
||||||
|
metadata:
|
||||||
|
name: flux-trust-store
|
||||||
|
namespace: flux-system
|
||||||
|
data:
|
||||||
|
known_hosts: |
|
||||||
|
github.com ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEmKSENjQEezOmxkZMy7opKgwFB9nkt5YRrYMjNuG5N87uRgg6CLrbo5wAdT/y6v0mKV0U2w0WZ2YB/++Tpockg=
|
||||||
|
```
|
||||||
|
|
||||||
|
Patch required on the main `kustomization.yaml`:
|
||||||
|
```yaml
|
||||||
|
- patch: |
|
||||||
|
- op: add
|
||||||
|
path: /spec/template/spec/containers/0/volumeMounts
|
||||||
|
value:
|
||||||
|
- name: ssh-trust-store
|
||||||
|
mountPath: /etc/ssh/ssh_known_hosts
|
||||||
|
subPath: known_hosts
|
||||||
|
readOnly: true
|
||||||
|
- op: add
|
||||||
|
path: /spec/template/spec/volumes
|
||||||
|
value:
|
||||||
|
- name: ssh-trust-store
|
||||||
|
configMap:
|
||||||
|
name: flux-trust-store
|
||||||
|
target:
|
||||||
|
kind: Deployment
|
||||||
|
name: "(kustomize|image-automation|source|image-reflector|helm|notification)-controller"
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Object-level Known Hosts Expansion
|
||||||
|
|
||||||
|
A new field is to be introduced into the existing kinds `ImageUpdateAutomation` and
|
||||||
|
`GitRepository`, to allow users to expand on the controller-level known hosts for
|
||||||
|
SSH operations:
|
||||||
|
```
|
||||||
|
spec:
|
||||||
|
trustStore:
|
||||||
|
ssh:
|
||||||
|
secretRef:
|
||||||
|
configMapRef:
|
||||||
|
```
|
||||||
|
|
||||||
|
The trust store can be expanded by either setting `spec.trustStore.ssh.secretRef` or
|
||||||
|
`spec.trustStore.ssh.configMapRef`, not both. Either option should contain the data
|
||||||
|
under a `known_hosts` key.
|
||||||
|
|
||||||
|
Known hosts configured this way will be aggregated with the ones defined at both
|
||||||
|
system and controller levels.
|
||||||
|
|
||||||
|
#### Pre-populated trust store
|
||||||
|
|
||||||
|
Flux container images would be pre-populated with [/etc/ssh/ssh_known_hosts] from
|
||||||
|
the main Git SaaS providers. As a result, users will only need to update their SSH
|
||||||
|
Trust Store for custom or less well known servers.
|
||||||
|
|
||||||
|
#### TLS
|
||||||
|
|
||||||
|
In TLS, the remote server identity is based on [public key infrastructure] and the
|
||||||
|
trust is based on the confirmation that the remote server's certificate was issued
|
||||||
|
by a "trusted" Certificate Authority (CA).
|
||||||
|
|
||||||
|
The OS level trust store contains the root trusted CAs, and any other certificate
|
||||||
|
that should be trusted by the machine. Note that CAs can verify other CAs, providing
|
||||||
|
an hierarchical chain of trust. Certificates that are not part of the chain, which
|
||||||
|
could be your own self-signed certificates, are considered untrustworthy by default.
|
||||||
|
TLS communications against untrusted remote servers are aborted.
|
||||||
|
|
||||||
|
#### Controller-level Trusted Certificates
|
||||||
|
|
||||||
|
**Note:** this requires no changes to the controllers, as this is based on the ways
|
||||||
|
in which TLS surface the trust store. This RFC only formalizes it as a supported
|
||||||
|
approach.
|
||||||
|
|
||||||
|
To trust CAs that are not part of the root trusted CAs, the OS level trust store
|
||||||
|
needs to be updated by mounting either a `ConfigMap` or `Secret` directly into the
|
||||||
|
Flux Controllers.
|
||||||
|
|
||||||
|
`Secret` example:
|
||||||
|
```yaml
|
||||||
|
apiVersion: v1
|
||||||
|
kind: Secret
|
||||||
|
metadata:
|
||||||
|
name: flux-trust-store
|
||||||
|
namespace: flux-system
|
||||||
|
data:
|
||||||
|
customCA.pem: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJoekNDQVMyZ0F3SUJBZ0lVZHNBdGlYM2dOMHVrN2RkeEFTV1lFL3RkdjB3d0NnWUlLb1pJemowRUF3SXcKR1RFWE1CVUdBMVVFQXhNT1pYaGhiWEJzWlM1amIyMGdRMEV3SGhjTk1qQXdOREUzTURneE9EQXdXaGNOTWpVdwpOREUyTURneE9EQXdXakFaTVJjd0ZRWURWUVFERXc1bGVHRnRjR3hsTG1OdmJTQkRRVEJaTUJNR0J5cUdTTTQ5CkFnRUdDQ3FHU000OUF3RUhBMElBQks3aC81RDhiVjkzTW1FZGh1MDJKc1M2dWdCOHM2UHpSbDNQVjR4czNTYnIKUk5ra001OSt4M2IwaVd4L2k3NnFQWXBOTG9pVlVWWFFtQTlZKzREYk14aWpVekJSTUE0R0ExVWREd0VCL3dRRQpBd0lCQmpBUEJnTlZIUk1CQWY4RUJUQURBUUgvTUIwR0ExVWREZ1FXQkJRR3lVaVUxUUVaaU1BcWpzbklZVHdaCjR5cDV3ekFQQmdOVkhSRUVDREFHaHdSL0FBQUJNQW9HQ0NxR1NNNDlCQU1DQTBnQU1FVUNJUUR6ZHR2S2RFOE8KMStXUlRaOU11U2lGWWNyRXo3Wm5lN1ZYb3VERUtxS0VpZ0lnTTRXbGJEZXVOQ0ticWhxait4WlYwcGEzcndlYgpPRDhFampDTVk2OVJNTzA9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
|
||||||
|
```
|
||||||
|
|
||||||
|
Patch required on the main `kustomization.yaml`:
|
||||||
|
```yaml
|
||||||
|
- patch: |
|
||||||
|
- op: add
|
||||||
|
path: /spec/template/spec/containers/0/volumeMounts
|
||||||
|
value:
|
||||||
|
- name: tls-trust-store
|
||||||
|
mountPath: /etc/ssl/certs/ca-cert-flux.pem
|
||||||
|
subPath: customCA.pem
|
||||||
|
readOnly: true
|
||||||
|
- op: add
|
||||||
|
path: /spec/template/spec/volumes
|
||||||
|
value:
|
||||||
|
- name: tls-trust-store
|
||||||
|
secret:
|
||||||
|
secretName: flux-trust-store
|
||||||
|
target:
|
||||||
|
kind: Deployment
|
||||||
|
name: "(kustomize|image-automation|source|image-reflector|helm|notification)-controller"
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Object-level Trusted Certificates Expansion
|
||||||
|
|
||||||
|
A new field is to be introduced into the existing kinds `Bucket`, `GitRepository`,
|
||||||
|
`HelmRepository`, `OCIRepository`, `ImageUpdateAutomation`, `Provider` and
|
||||||
|
`ImageRepository`, to allow users to expand on trusted CAs at controller-level for
|
||||||
|
HTTPS operations:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
spec:
|
||||||
|
trustStore:
|
||||||
|
tls:
|
||||||
|
secretRef:
|
||||||
|
configMapRef:
|
||||||
|
```
|
||||||
|
|
||||||
|
The trust store can be expanded by either setting `spec.trustStore.tls.secretRef`
|
||||||
|
or `spec.trustStore.tls.configMapRef`, not both. Either option should contain the
|
||||||
|
data under a `caFile` key.
|
||||||
|
|
||||||
|
CA bundles configured this way will be aggregated with the ones defined at both
|
||||||
|
system and controller levels.
|
||||||
|
|
||||||
|
#### Pre-populated trust store
|
||||||
|
|
||||||
|
Flux container images already come with pre-populated CA roots, which are
|
||||||
|
automatically updated by the Linux distribution used on the base images.
|
||||||
|
As a result, users only need to update their TLS Trust Store when acessing
|
||||||
|
web servers using certificates that were not signed by a Publicly trusted CA.
|
||||||
|
|
||||||
|
### Enabling Object-Level Trust Store
|
||||||
|
|
||||||
|
Object-level trust store expansion is disabled by default. To enable it start
|
||||||
|
the controller with:
|
||||||
|
|
||||||
|
`--insecure-object-trust-store={tls-only,ssh-only,both}`
|
||||||
|
|
||||||
|
The flag defaults to `disabled`:
|
||||||
|
`--insecure-object-trust-store={disabled}`
|
||||||
|
|
||||||
|
### User Stories
|
||||||
|
|
||||||
|
#### Story 1
|
||||||
|
|
||||||
|
> As a tenant, I want to be able to expand trust settings so that I can
|
||||||
|
> connect to my own servers without needing to ask an administrator.
|
||||||
|
|
||||||
|
#### Story 2
|
||||||
|
|
||||||
|
> As a Platform admin, I want to configure all trusted servers at the controller
|
||||||
|
> level and block any specific team from overriding those settings.
|
||||||
|
|
||||||
|
#### Story 3
|
||||||
|
|
||||||
|
> As a Security Auditor, I want to be able to review all Known Hosts and CA Bundles
|
||||||
|
> being used within a Flux instance, without requiring RBAC access to more sensitive
|
||||||
|
> information.
|
||||||
|
|
||||||
|
### Alternatives
|
||||||
|
|
||||||
|
#### Consume controller-level settings via two new flags
|
||||||
|
|
||||||
|
Two new flags would be added into the controllers (`--tls-ca-bundles-secret` and
|
||||||
|
`--ssh-known-hosts-secret`) allowing for secrets to be consumed at startup time.
|
||||||
|
|
||||||
|
This would establish a "flux-specific" approach, which would not be aligned with
|
||||||
|
existing tools and applications that may need to coexist in the same container,
|
||||||
|
meaning that a Flux controller may trust a server, whilst other applications within
|
||||||
|
the container would not - or vice-versa.
|
||||||
|
|
||||||
|
#### Remove object-level trust store settings
|
||||||
|
|
||||||
|
Instead of creating a toggle to disable object-level trust settings, the entire
|
||||||
|
feature could have been deprecated. We have decided that by keeping the feature
|
||||||
|
in would allow for an easier transition.
|
||||||
|
|
||||||
|
#### Skip the implementation of the object-level blocker
|
||||||
|
|
||||||
|
Instead of creating a built-in feature to block the use of object-level Trust
|
||||||
|
Store expansion, we could rely on other tools and mechanisms within the Kubernetes
|
||||||
|
ecosystem (e.g. OPA) to enable users to achieve the same outcome.
|
||||||
|
|
||||||
|
Due to the importance that Flux has in the bootstrapping of clusters, such an
|
||||||
|
important requirement (enforce trust at controller level) should be inherit to
|
||||||
|
the controllers, instead of delegated to third party components.
|
||||||
|
|
||||||
|
## Design Details
|
||||||
|
|
||||||
|
### Auto-populating SSH Trust Store
|
||||||
|
|
||||||
|
Flux container images that access Git SSH servers (e.g. Source Controller, Image
|
||||||
|
Automation Controller and Flux CLI) will contain entries on [/etc/ssh/ssh_known_hosts]
|
||||||
|
for the most popular Git SaaS providers.
|
||||||
|
|
||||||
|
Each provider will contain one entry for each supported host key algorithm.
|
||||||
|
The `ssh_known_hosts` will be a static file in the respective repositories, and
|
||||||
|
the Dockerfile will simply copy it into the final image.
|
||||||
|
|
||||||
|
The known hosts will be updated via automation, which will issue PRs for the maintainers
|
||||||
|
to review and then approve. As a result, the trusted known hosts will be deterministic
|
||||||
|
based on the container image version used, in the same way that CAs are.
|
||||||
|
|
||||||
|
### Refreshing Controller-level Trust Store Values
|
||||||
|
|
||||||
|
The proposed approach heavily relies on built-in functionality in Kubernetes
|
||||||
|
and Linux distributions. Therefore, the disk contents will be automatically
|
||||||
|
refreshed when either [Secrets] or [ConfigMaps] are changed.
|
||||||
|
|
||||||
|
All SSH operations would need to read the file again for each operation, which
|
||||||
|
is analogous to the existing "load from memory" approach in place.
|
||||||
|
|
||||||
|
For TLS, this value is cached on first use and won't be refreshed until the
|
||||||
|
controller is restarted. In some instances, the recurrent failure by the
|
||||||
|
controller to establish connections with a remote server could cause the Pod
|
||||||
|
to be restarted, resulting in the TLS certs being refreshed.
|
||||||
|
|
||||||
|
[Secrets]: https://kubernetes.io/docs/concepts/configuration/secret/#mounted-secrets-are-updated-automatically
|
||||||
|
[ConfigMaps]: https://kubernetes.io/docs/concepts/configuration/configmap/#mounted-configmaps-are-updated-automatically
|
||||||
|
|
||||||
|
### CA Trust Location and Auto Discovery
|
||||||
|
|
||||||
|
**Note:** this requires no changes to the controllers. The below only calls out
|
||||||
|
the existing Go standard library behavior.
|
||||||
|
|
||||||
|
The CA Trust Store location `/etc/ssl/certs/` referenced here is the default
|
||||||
|
location in Alpine distros, which is what is currently used across all Flux
|
||||||
|
images. Users can use other default locations, as per defined in the [Go standard library].
|
||||||
|
Another option is to define a custom CA Trust Store via [SSL_CERT_DIR].
|
||||||
|
|
||||||
|
On first Transport creation, Go will load any bundled `.crt` files and then
|
||||||
|
append any unique `.pem` files which are inside the certificate directory.
|
||||||
|
Therefore, from a Go perspective, new `.pem` files will be taken into account,
|
||||||
|
even when they are not bundled into the default `/etc/ssl/certs/ca-certificates.crt`.
|
||||||
|
|
||||||
|
[Go standard library]: https://github.com/golang/go/blob/master/src/crypto/x509/root_linux.go#L18
|
||||||
|
[SSL_CERT_DIR]: https://github.com/golang/go/blob/master/src/crypto/x509/root_unix.go#L53
|
||||||
|
|
||||||
|
### SSH and TLS references
|
||||||
|
|
||||||
|
The new fields `spec.trustStore.tls` and `spec.trustStore.ssh` analogous
|
||||||
|
to Kubernetes `EnvFromSource`, in which it can be used to define either a
|
||||||
|
`configMapRef` or a `secretRef`, but not both.
|
||||||
|
|
||||||
|
## Implementation History
|
||||||
|
|
||||||
|
<!--
|
||||||
|
Major milestones in the lifecycle of the RFC such as:
|
||||||
|
- The first Flux release where an initial version of the RFC was available.
|
||||||
|
- The version of Flux where the RFC graduated to general availability.
|
||||||
|
- The version of Flux where the RFC was retired or superseded.
|
||||||
|
-->
|
||||||
|
|
||||||
|
[/etc/ssh/ssh_known_hosts]: https://en.wikibooks.org/wiki/OpenSSH/Client_Configuration_Files#/etc/ssh/ssh_known_hosts
|
||||||
|
[public key infrastructure]: https://en.wikipedia.org/wiki/Public_key_infrastructure
|
||||||
|
[Trust on first use]: https://en.wikipedia.org/wiki/Trust_on_first_use
|
||||||
Loading…
Reference in New Issue