1
0
mirror of synced 2026-02-13 13:06:56 +00:00

make tests for notifications provider agnostic

Signed-off-by: Somtochi Onyekwere <somtochionyekwere@gmail.com>
This commit is contained in:
Somtochi Onyekwere
2023-06-27 10:34:07 +01:00
committed by Sunny
parent 7c1b897919
commit e63ddb99de
18 changed files with 515 additions and 287 deletions

View File

@@ -1,7 +1,7 @@
# E2E Tests
The goal is to verify that Flux integration with cloud providers are actually working now and in the future.
Currently, we only have tests for Azure.
Currently, we only have tests for Azure and GCP.
## General requirements
@@ -60,9 +60,10 @@ the tests:
### Architecture
The [gcp](./terraform/gcp) terraform files create the GKE cluster and related resources to run the tests. It creates:
- An Google Container Registry and Artifact Registry
- An Google Kubernetes Cluster
- A Google Container Registry and Artifact Registry
- A Google Kubernetes Cluster
- Two Google Cloud Source Repositories
- A Google Pub/Sub Topic and a subscription to the service that would be used in the tests
Note: It doesn't create Google KMS keyrings and crypto keys because these cannot be destroyed. Instead, you have
to pass in the crypto key and keyring that would be used to test the sops encryption in Flux. Please see `.env.sample`
@@ -72,6 +73,8 @@ for the terraform variables
- GCP account with an active project to be able to create GKE and GCR, and permission to assign roles.
- Existing GCP KMS keyring and crypto key.
- [Create a Keyring](https://cloud.google.com/kms/docs/create-key-ring)
- [Create a Crypto Key](https://cloud.google.com/kms/docs/create-key)
- gcloud CLI, need to be logged in using `gcloud auth login` as a User (not a
Service Account), configure application default credentials with `gcloud auth
application-default login` and docker credential helper with `gcloud auth configure-docker`.
@@ -86,7 +89,6 @@ for the terraform variables
docker accordingly
```console
$ gcloud auth print-access-token | docker login -u oauth2accesstoken --password-stdin https://us-central1-docker.pkg.dev
$ gcloud auth print-access-token | docker login -u oauth2accesstoken --password-stdin https://gcr.io
```
In this case, the GCP client in terraform uses the Service Account to
authenticate and the gcloud CLI is used only to authenticate with Google
@@ -100,15 +102,33 @@ for the terraform variables
to prevent aggressive masking in the logs. Refer
[aggressive replacement in logs](https://github.com/google-github-actions/auth/blob/v1.1.0/docs/TROUBLESHOOTING.md#aggressive--replacement-in-logs)
for more details.
- Register SSH Keys with Google Cloud. # add docs
- Register [SSH Keys with Google Cloud](https://cloud.google.com/source-repositories/docs/authentication#ssh)
- Google Cloud supports these three SSH key types: RSA (only for keys with more than 2048 bits), ECDSA and ED25519
- **Note:** Google doesn't allow an SSH key to be associated with a service account email address. Therefore, there has to be an actual
user that the SSH keys are registered to, and the email of this user will be passed to terraform through the `TF_VAR_gcp_email`
variable.
### Permissions
Following roles are needed for provisioning the infrastructure and running the tests:
- Compute Instance Admin (v1)
- Kubernetes Engine Admin
- Service Account User
- Artifact Registry Administrator
- Artifact Registry Repository Administrator
- Cloud KMS Admin
- Cloud KMS CryptoKey Encrypter
- Source Repository Administrator
- Pub/Sub Admin
## Tests
Each test run is initiated by running `terraform apply` in the provider's terraform directory e.g terraform apply,
it does this by using the [tftestenv package](https://github.com/fluxcd/test-infra/blob/main/tftestenv/testenv.go)
within the `fluxcd/test-infra` repository. It then reads the output of the Terraform to get information needed
for the tests like the kubernetes client ID, the azure DevOps repository urls, the key vault ID etc. This means that
a lot of the communication with the Azure API is offset to Terraform instead of requiring it to be implemented in the test.
for the tests like the kubernetes client ID, the cloud repository urls, the key vault ID etc. This means that
a lot of the communication with the cloud provider API is offset to Terraform instead of requiring it to be implemented in the test.
The following tests are currently implemented:
@@ -118,11 +138,11 @@ The following tests are currently implemented:
- kustomize-controller can decrypt secrets using SOPS and provider key vault
- image-automation-controller can create branches and push to cloud repositories (https+ssh)
- source-controller can pull charts from cloud provider container registry Helm repositories
- notification-controller can forward events to cloud Events Service(EventHub for Azure and Google Pub/Sub)
The following tests are run only for Azure since it is supported in the notification-controller:
- notification-controller can send commit status to Azure DevOps
- notification-controller can forward events to Azure Event Hub
### Running tests locally
@@ -165,8 +185,9 @@ ok github.com/fluxcd/flux2/tests/integration 947.341s
In the above, the test created a build directory build/ and the flux cli binary is copied build/flux. It would be used
to bootstrap Flux on the cluster. You can configure the location of the Flux CLI binary by setting the FLUX_BINARY variable.
We also pull two version of `ghcr.io/stefanprodan/podinfo` image. These images are pushed to the Azure Container Registry
and used to test `ImageRepository` and `ImageUpdateAutomation`. The terraform resources get created and the tests are run.
We also pull two version of `ghcr.io/stefanprodan/podinfo` image. These images are pushed to the cloud provider's
Container Registry and used to test `ImageRepository` and `ImageUpdateAutomation`. The terraform resources get created
and the tests are run.
**IMPORTANT:** In case the terraform infrastructure results in a bad state, maybe due to a crash during the apply,
the whole infrastructure can be destroyed by running terraform destroy in terraform/<provider> directory.