flux2/rfcs/0000-trust-store-for-tls-and-ssh

RFC-NNNN Trust Store for TLS and SSH

Status: provisional

Creation date: 2022-12-02

Last update: 2022-12-02

Summary

Consolidates and formalizes the supported ways to establish trusted connections with remote servers via Transport Layer Security (TLS) and Secure Shell (SSH). Resulting on new ways to configure trust, and allowing administrators the capability to disable some of the existing options.

Motivation

The current model could be improved by allowing for controller-level trust configurations, so that multiple objects connecting to the same server don't need to specify overrides. The approach aligns with both TLS and SSH canonical OS level implementations, in which they rely on a global trust store to define machine level trust, but users and applications can further expand on trusted servers (or CAs), when not blocked by administrators.

Known Hosts (used by SSH connections), and CA Bundles (for TLS), are not particularly sensitive information - when leaving aside privacy considerations. Before this RFC, the officially supported approach leans on secrets to pass this information to the controllers. The same secret is also use to provide user credentials, which is more sensitive in nature, making this sub-optimal from a security stand-point.

Goals

• Consolidate the officially supported trust settings across the Flux ecosystem.
• Formalize support for configuring trust at controller-level.
• Add toggle to block object-level trust overrides.
• Enable users to surface trust information securely.

Non-Goals

• Maintain backwards compatibility with older versions of Flux.

Proposal

For configuring system-wide trust, Flux would rely on the well-established OS-level trust store. When dynamically mounting of the trust store is required, it will be enabled by using Kubernetes Secret and ConfigMap mounting. When immutable trust store is required, users can build their own version of the controllers, with their baked-in settings.

TLS and SSH use different techniques to establish the identity of remote servers, each relying on its own trust store.

The sections below will dive into the specifics of each one, highlighting their details, changes required and example of the proposed usage.

A new way to configure object-level Trust Store overrides is also being proposed, in combination with a controller level toggle to disable it.

SSH

In SSH, the remote server identity is based on Trust on first use. At first connection to a new server, the user confirms whether or not to trust that server based on the server's Public key fingerprint.

In the context of Flux, which provides no user interaction, if the remote server finger print is not configured within the provided set of Known Hosts, the connection is aborted.

Controller-level Known Hosts

For setting controller level Known Hosts, we propose the use of the existing Linux file in disk: /etc/ssh/ssh_known_hosts.

Users would be able to configure the OS level trust store by mounting either a ConfigMap or Secret directly into the Flux Controllers.

ConfigMap example:

apiVersion: v1
kind: ConfigMap
metadata:
  name: flux-trust-store
  namespace: flux-system
data:
  known_hosts: |
        github.com ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEmKSENjQEezOmxkZMy7opKgwFB9nkt5YRrYMjNuG5N87uRgg6CLrbo5wAdT/y6v0mKV0U2w0WZ2YB/++Tpockg=

Patch required on the main kustomization.yaml:

  - patch: |
      - op: add
        path: /spec/template/spec/containers/0/volumeMounts
        value:
          - name: ssh-trust-store
            mountPath: /etc/ssh/ssh_known_hosts
            subPath: known_hosts
            readOnly: true
      - op: add
        path: /spec/template/spec/volumes
        value:
          - name: ssh-trust-store
            configMap:
              name: flux-trust-store      
    target:
      kind: Deployment
      name: "(kustomize|image-automation|source|image-reflector|helm|notification)-controller"

Object-level Known Hosts Expansion

A new field is to be introduced into the existing kinds ImageUpdateAutomation and GitRepository, to allow users to expand on the controller-level known hosts for SSH operations:

spec:
  trustStore:
    ssh:
        secretRef:
        configMapRef:

The trust store can be expanded by either setting spec.trustStore.ssh.secretRef or spec.trustStore.ssh.configMapRef, not both. Either option should contain the data under a known_hosts key.

Known hosts configured this way will be aggregated with the ones defined at both system and controller levels.

TLS

In TLS, the remote server identity is based on public key infrastructure and the trust is based on the confirmation that the remote server's certificate was issued by a "trusted" Certificate Authority (CA).

The OS level trust store contains the root trusted CAs, and any other certificate that should be trusted by the machine. Note that CAs can verify other CAs, providing an hierarchical chain of trust. Certificates that are not part of the chain, which could be your own self-signed certificates, are considered untrustworthy by default. TLS communications against untrusted remote servers are aborted.

Controller-level Trusted Certificates

NOTE: this requires no changes on the controllers, as this is based on the ways that TLS surface the trust store. This RFC only formalizes it as a supported approach.

To trust CAs that are not part of the root trusted CAs, the OS level trust store needs to be updated by mounting either a ConfigMap or Secret directly into the Flux Controllers.

Secret example:

apiVersion: v1
kind: Secret
metadata:
  name: flux-trust-store
  namespace: flux-system
data:
  customCA.pem: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJoekNDQVMyZ0F3SUJBZ0lVZHNBdGlYM2dOMHVrN2RkeEFTV1lFL3RkdjB3d0NnWUlLb1pJemowRUF3SXcKR1RFWE1CVUdBMVVFQXhNT1pYaGhiWEJzWlM1amIyMGdRMEV3SGhjTk1qQXdOREUzTURneE9EQXdXaGNOTWpVdwpOREUyTURneE9EQXdXakFaTVJjd0ZRWURWUVFERXc1bGVHRnRjR3hsTG1OdmJTQkRRVEJaTUJNR0J5cUdTTTQ5CkFnRUdDQ3FHU000OUF3RUhBMElBQks3aC81RDhiVjkzTW1FZGh1MDJKc1M2dWdCOHM2UHpSbDNQVjR4czNTYnIKUk5ra001OSt4M2IwaVd4L2k3NnFQWXBOTG9pVlVWWFFtQTlZKzREYk14aWpVekJSTUE0R0ExVWREd0VCL3dRRQpBd0lCQmpBUEJnTlZIUk1CQWY4RUJUQURBUUgvTUIwR0ExVWREZ1FXQkJRR3lVaVUxUUVaaU1BcWpzbklZVHdaCjR5cDV3ekFQQmdOVkhSRUVDREFHaHdSL0FBQUJNQW9HQ0NxR1NNNDlCQU1DQTBnQU1FVUNJUUR6ZHR2S2RFOE8KMStXUlRaOU11U2lGWWNyRXo3Wm5lN1ZYb3VERUtxS0VpZ0lnTTRXbGJEZXVOQ0ticWhxait4WlYwcGEzcndlYgpPRDhFampDTVk2OVJNTzA9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K

Patch required on the main kustomization.yaml:

  - patch: |
      - op: add
        path: /spec/template/spec/containers/0/volumeMounts
        value:
          - name: tls-trust-store
            mountPath: /etc/ssl/certs/ca-cert-flux.pem
            subPath: customCA.pem
            readOnly: true
      - op: add
        path: /spec/template/spec/volumes
        value:
          - name: tls-trust-store
            secret:
              secretName: flux-trust-store      
    target:
      kind: Deployment
      name: "(kustomize|image-automation|source|image-reflector|helm|notification)-controller"

Object-level Trusted Certificates Expansion

A new field is to be introduced into the existing kinds Bucket, GitRepository, HelmRepository, OCIRepository, ImageUpdateAutomation, Provider and ImageRepository, to allow users to expand on the controller-level known hosts for SSH operations:

spec:
  trustStore:
    tls:
        secretRef:
        configMapRef:

The trust store can be expanded by either setting spec.trustStore.tls.secretRef or spec.trustStore.tls.configMapRef, not both. Either option should contain the data under a caFile key.

CA bundles configured this way will be aggregated with the ones defined at both system and controller levels.

Enabling Object-Level Trust Store

Object-level trust store expansion is disabled by default. To enable it start the controller with:

--insecure-object-trust-store={tls-only,ssh-only,both}

The flag defaults to disabled: --insecure-object-trust-store={disabled}

User Stories

Story 1

As a tenant, I want to be able to expand trust settings so that I can connect to my own servers without needing to ask an administrator.

Story 2

As a Platform admin, I want to configure all trusted servers at the controller level and block any specific team from overriding those settings.

Story 3

As a Security Auditor, I want to be able to review all Known Hosts and CA Bundles being used within a Flux instance, without requiring RBAC access to more sensitive information.

Alternatives

Consume controller-level settings via two new flags

Two new flags would be added into the controllers (--tls-ca-bundles-secret and --ssh-known-hosts-secret) allowing for secrets to be consumed at startup time.

This would establish a "flux-specific" approach, which would not be aligned with existing tools and applications that may need to coexist in the same container, meaning that a Flux controller may trust a server, whilst other applications within the container would not - or vice-versa.

Remove object-level trust store settings

Instead of creating a toggle to disable object-level trust settings, the entire feature could have been deprecated. We have decided that by keeping the feature in would allow for an easier transition.

Skip the implementation of the object-level blocker

Instead of creating a built-in feature to block the use of object-level Trust Store expansion, we could rely on other tools and mechanisms within the Kubernetes ecosystem (e.g. OPA) to enable users to achieve the same outcome.

Due to the importance that Flux has in the bootstrapping of clusters, such an important requirement (enforce trust at controller level) should be inherit to the controllers, instead of delegated to third party components.

Design Details

Refreshing Trust Store Values

NOTE: Section still WIP.

Values are automatically refreshed from Secrets and ConfigMaps into disk. SSH would need to read the file again for each operation. TLS would be automatically refreshed via Transport level system roots.

https://kubernetes.io/docs/concepts/configuration/secret/#mounted-secrets-are-updated-automatically https://kubernetes.io/docs/concepts/configuration/configmap/#mounted-configmaps-are-updated-automatically

SSH and TLS references

NOTE: Section still WIP.

Analogous to Kubernetes' EnvFromSource, in which it can source either a ConfigMap or a Secret.

Implementation History