Merge pull request #4017 from somtochiama/gcp-e2e

Add e2e tests for GCP

tests/integration/.env.sample

@@ -1,11 +1,7 @@
 ## Azure
-export TF_VAR_azuredevops_org=
+# export TF_VAR_azuredevops_org=
-export TF_VAR_azuredevops_pat=
+# export TF_VAR_azuredevops_pat=
-export TF_VAR_azure_location=
+# export TF_VAR_azure_location=
-## These are not terraform variables
-## but they are needed for the azure tests
-export AZUREDEVOPS_SSH=
-export AZUREDEVOPS_SSH_PUB=
 ## Set the following only when authenticating using Service Principal (suited
 ## for CI environment).
 # export ARM_CLIENT_ID=
@@ -13,5 +9,21 @@ export AZUREDEVOPS_SSH_PUB=
 # export ARM_SUBSCRIPTION_ID=
 # export ARM_TENANT_ID=
 
+## GCP
+# export TF_VAR_gcp_project_id=
+# export TF_VAR_gcp_zone=
+# export TF_VAR_gcp_region=
+# export TF_VAR_gcp_keyring=
+# export TF_VAR_gcp_crypto_key=
+## Email address of a GCP user used for git repository cloning over ssh.
+# export TF_VAR_gcp_email=
+## Set the following only when using service account.
+## Provide absolute path to the service account JSON key file.
+# export GOOGLE_APPLICATION_CREDENTIALS=
+
 ## Common variables
 # export TF_VAR_tags='{"environment"="dev", "createdat"='"\"$(date -u +x%Y-%m-%d_%Hh%Mm%Ss)\""'}'
+## These are not terraform variables
+## but they are needed for the bootstrap tests
+# export GITREPO_SSH_PATH=
+# export GITREPO_SSH_PUB_PATH=

tests/integration/README.md

@@ -1,7 +1,7 @@
 # E2E Tests
 
 The goal is to verify that Flux integration with cloud providers are actually working now and in the future.
-Currently, we only have tests for Azure.
+Currently, we only have tests for Azure and GCP.
 
 ## General requirements
 
@@ -55,14 +55,80 @@ the tests:
 - `Microsoft.KeyVault/*`
 - `Microsoft.EventHub/*`
 
+## GCP
+
+### Architecture
+
+The [gcp](./terraform/gcp) terraform files create the GKE cluster and related resources to run the tests. It creates:
+- A Google Container Registry and Artifact Registry
+- A Google Kubernetes Cluster
+- Two Google Cloud Source Repositories
+- A Google Pub/Sub Topic and a subscription to the service that would be used in the tests
+
+Note: It doesn't create Google KMS keyrings and crypto keys because these cannot be destroyed. Instead, you have
+to pass in the crypto key and keyring that would be used to test the sops encryption in Flux. Please see `.env.sample`
+for the terraform variables
+
+### Requirements
+
+- GCP account with an active project to be able to create GKE and GCR, and permission to assign roles.
+- Existing GCP KMS keyring and crypto key.
+  - [Create a Keyring](https://cloud.google.com/kms/docs/create-key-ring)
+  - [Create a Crypto Key](https://cloud.google.com/kms/docs/create-key)
+- gcloud CLI, need to be logged in using `gcloud auth login` as a User (not a
+  Service Account), configure application default credentials with `gcloud auth
+  application-default login` and docker credential helper with `gcloud auth configure-docker`.
+
+  **NOTE:** To use Service Account (for example in CI environment), set
+  `GOOGLE_APPLICATION_CREDENTIALS` variable in `.env` with the path to the JSON
+  key file, source it and authenticate gcloud CLI with:
+  ```console
+  $ gcloud auth activate-service-account --key-file=$GOOGLE_APPLICATION_CREDENTIALS
+  ```
+  Depending on the Container/Artifact Registry host used in the test, authenticate
+  docker accordingly
+  ```console
+  $ gcloud auth print-access-token | docker login -u oauth2accesstoken --password-stdin https://us-central1-docker.pkg.dev
+  ```
+  In this case, the GCP client in terraform uses the Service Account to
+  authenticate and the gcloud CLI is used only to authenticate with Google
+  Container Registry and Google Artifact Registry.
+
+  **NOTE FOR CI USAGE:** When saving the JSON key file as a CI secret, compress
+  the file content with
+  ```console
+  $ cat key.json | jq -r tostring
+  ```
+  to prevent aggressive masking in the logs. Refer
+  [aggressive replacement in logs](https://github.com/google-github-actions/auth/blob/v1.1.0/docs/TROUBLESHOOTING.md#aggressive--replacement-in-logs)
+  for more details.
+- Register [SSH Keys with Google Cloud](https://cloud.google.com/source-repositories/docs/authentication#ssh)
+  - Google Cloud supports these three SSH key types: RSA (only for keys with more than 2048 bits), ECDSA and ED25519
+  - **Note:** Google doesn't allow an SSH key to be associated with a service account email address. Therefore, there has to be an actual
+    user that the SSH keys are registered to, and the email of this user will be passed to terraform through the `TF_VAR_gcp_email`
+    variable.
+
+### Permissions
+
+Following roles are needed for provisioning the infrastructure and running the tests:
+
+- Compute Instance Admin (v1)
+- Kubernetes Engine Admin
+- Service Account User
+- Artifact Registry Administrator
+- Artifact Registry Repository Administrator
+- Cloud KMS Admin
+- Cloud KMS CryptoKey Encrypter
+- Source Repository Administrator
+- Pub/Sub Admin
 
 ## Tests
 
 Each test run is initiated by running `terraform apply` in the provider's terraform directory e.g terraform apply,
 it does this by using the [tftestenv package](https://github.com/fluxcd/test-infra/blob/main/tftestenv/testenv.go)
 within the `fluxcd/test-infra` repository. It then reads the output of the Terraform to get information needed
-for the tests like the kubernetes client ID, the azure DevOps repository urls, the key vault ID etc. This means that
+for the tests like the kubernetes client ID, the cloud repository urls, the key vault ID etc. This means that
-a lot of the communication with the Azure API is offset to Terraform instead of requiring it to be implemented in the test.
+a lot of the communication with the cloud provider API is offset to Terraform instead of requiring it to be implemented in the test.
 
 The following tests are currently implemented:
 
@@ -72,11 +138,11 @@ The following tests are currently implemented:
 - kustomize-controller can decrypt secrets using SOPS and provider key vault
 - image-automation-controller can create branches and push to cloud repositories (https+ssh)
 - source-controller can pull charts from cloud provider container registry Helm repositories
+- notification-controller can forward events to cloud Events Service(EventHub for Azure and Google Pub/Sub)
 
 The following tests are run only for Azure since it is supported in the notification-controller:
 
 - notification-controller can send commit status to Azure DevOps
-- notification-controller can forward events to Azure Event Hub
 
 ### Running tests locally
 
@@ -119,8 +185,9 @@ ok      github.com/fluxcd/flux2/tests/integration       947.341s
 
 In the above, the test created a build directory build/ and the flux cli binary is copied build/flux. It would be used
 to bootstrap Flux on the cluster. You can configure the location of the Flux CLI binary by setting the FLUX_BINARY variable.
-We also pull two version of `ghcr.io/stefanprodan/podinfo` image. These images are pushed to the Azure Container Registry
+We also pull two version of `ghcr.io/stefanprodan/podinfo` image. These images are pushed to the cloud provider's
-and used to test `ImageRepository` and `ImageUpdateAutomation`. The terraform resources get created and the tests are run.
+Container Registry and used to test `ImageRepository` and `ImageUpdateAutomation`. The terraform resources get created
+and the tests are run.
 
 **IMPORTANT:** In case the terraform infrastructure results in a bad state, maybe due to a crash during the apply,
 the whole infrastructure can be destroyed by running terraform destroy in terraform/<provider> directory.

tests/integration/azure_specific_test.go

@@ -21,7 +21,6 @@ package integration
 
 import (
 	"context"
-	"encoding/json"
 	"fmt"
 	"io"
 	"log"
@@ -29,7 +28,6 @@ import (
 	"testing"
 	"time"
 
-	eventhub "github.com/Azure/azure-event-hubs-go/v3"
 	"github.com/microsoft/azure-devops-go-api/azuredevops"
 	"github.com/microsoft/azure-devops-go-api/azuredevops/git"
 	. "github.com/onsi/gomega"
@@ -37,185 +35,14 @@ import (
 
 	corev1 "k8s.io/api/core/v1"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
-	"k8s.io/apimachinery/pkg/types"
 
 	kustomizev1 "github.com/fluxcd/kustomize-controller/api/v1"
 	notiv1 "github.com/fluxcd/notification-controller/api/v1"
 	notiv1beta2 "github.com/fluxcd/notification-controller/api/v1beta2"
-	events "github.com/fluxcd/pkg/apis/event/v1beta1"
 	"github.com/fluxcd/pkg/apis/meta"
 	sourcev1 "github.com/fluxcd/source-controller/api/v1"
 )
 
-func TestEventHubNotification(t *testing.T) {
-	g := NewWithT(t)
-
-	ctx := context.TODO()
-	branchName := "test-notification"
-	testID := branchName + "-" + randStringRunes(5)
-
-	// Start listening to eventhub with latest offset
-	// TODO(somtochiama): Make here provider agnostic
-	hub, err := eventhub.NewHubFromConnectionString(cfg.notificationURL)
-	g.Expect(err).ToNot(HaveOccurred())
-	c := make(chan string, 10)
-	handler := func(ctx context.Context, event *eventhub.Event) error {
-		c <- string(event.Data)
-		return nil
-	}
-	runtimeInfo, err := hub.GetRuntimeInformation(ctx)
-	g.Expect(err).ToNot(HaveOccurred())
-	g.Expect(len(runtimeInfo.PartitionIDs)).To(Equal(1))
-	listenerHandler, err := hub.Receive(ctx, runtimeInfo.PartitionIDs[0], handler, eventhub.ReceiveWithLatestOffset())
-	g.Expect(err).ToNot(HaveOccurred())
-
-	// Setup Flux resources
-	manifest := `apiVersion: v1
-kind: ConfigMap
-metadata:
-  name: foobar`
-	repoUrl := getTransportURL(cfg.applicationRepository)
-	client, err := getRepository(ctx, t.TempDir(), repoUrl, defaultBranch, cfg.defaultAuthOpts)
-	g.Expect(err).ToNot(HaveOccurred())
-	files := make(map[string]io.Reader)
-	files["configmap.yaml"] = strings.NewReader(manifest)
-	err = commitAndPushAll(ctx, client, files, branchName)
-	g.Expect(err).ToNot(HaveOccurred())
-
-	namespace := corev1.Namespace{
-		ObjectMeta: metav1.ObjectMeta{
-			Name: testID,
-		},
-	}
-	g.Expect(testEnv.Create(ctx, &namespace)).To(Succeed())
-	defer testEnv.Delete(ctx, &namespace)
-
-	secret := corev1.Secret{
-		ObjectMeta: metav1.ObjectMeta{
-			Name:      testID,
-			Namespace: testID,
-		},
-		StringData: map[string]string{
-			"address": cfg.notificationURL,
-		},
-	}
-	g.Expect(testEnv.Create(ctx, &secret)).To(Succeed())
-	defer testEnv.Delete(ctx, &secret)
-
-	provider := notiv1beta2.Provider{
-		ObjectMeta: metav1.ObjectMeta{
-			Name:      testID,
-			Namespace: testID,
-		},
-		Spec: notiv1beta2.ProviderSpec{
-			Type:    "azureeventhub",
-			Address: repoUrl,
-			SecretRef: &meta.LocalObjectReference{
-				Name: testID,
-			},
-		},
-	}
-	g.Expect(testEnv.Create(ctx, &provider)).To(Succeed())
-	defer testEnv.Delete(ctx, &provider)
-
-	alert := notiv1beta2.Alert{
-		ObjectMeta: metav1.ObjectMeta{
-			Name:      testID,
-			Namespace: testID,
-		},
-		Spec: notiv1beta2.AlertSpec{
-			ProviderRef: meta.LocalObjectReference{
-				Name: provider.Name,
-			},
-			EventSources: []notiv1.CrossNamespaceObjectReference{
-				{
-					Kind:      "Kustomization",
-					Name:      testID,
-					Namespace: testID,
-				},
-			},
-		},
-	}
-	g.Expect(testEnv.Create(ctx, &alert)).ToNot(HaveOccurred())
-	defer testEnv.Delete(ctx, &alert)
-
-	g.Eventually(func() bool {
-		nn := types.NamespacedName{Name: alert.Name, Namespace: alert.Namespace}
-		alertObj := &notiv1beta2.Alert{}
-		err := testEnv.Get(ctx, nn, alertObj)
-		if err != nil {
-			return false
-		}
-		if err := checkReadyCondition(alertObj); err != nil {
-			t.Log(err)
-			return false
-		}
-
-		return true
-	}, testTimeout, testInterval).Should(BeTrue())
-
-	modifyKsSpec := func(spec *kustomizev1.KustomizationSpec) {
-		spec.Interval = metav1.Duration{Duration: 30 * time.Second}
-		spec.HealthChecks = []meta.NamespacedObjectKindReference{
-			{
-				APIVersion: "v1",
-				Kind:       "ConfigMap",
-				Name:       "foobar",
-				Namespace:  testID,
-			},
-		}
-	}
-	g.Expect(setUpFluxConfig(ctx, testID, nsConfig{
-		repoURL: repoUrl,
-		ref: &sourcev1.GitRepositoryRef{
-			Branch: branchName,
-		},
-		path:         "./",
-		modifyKsSpec: modifyKsSpec,
-	})).To(Succeed())
-	t.Cleanup(func() {
-		err := tearDownFluxConfig(ctx, testID)
-		if err != nil {
-			t.Logf("failed to delete resources in '%s' namespace: %s", testID, err)
-		}
-	})
-
-	g.Eventually(func() bool {
-		err := verifyGitAndKustomization(ctx, testEnv, testID, testID)
-		if err != nil {
-			t.Log(err)
-			return false
-		}
-		return true
-	}, testTimeout, testInterval).Should(BeTrue())
-
-	// Wait to read even from event hub
-	g.Eventually(func() bool {
-		select {
-		case eventJson := <-c:
-			event := &events.Event{}
-			err := json.Unmarshal([]byte(eventJson), event)
-			if err != nil {
-				t.Logf("the received event type does not match Flux format, error: %v", err)
-				return false
-			}
-
-			if event.InvolvedObject.Kind == kustomizev1.KustomizationKind &&
-				event.InvolvedObject.Name == testID && event.InvolvedObject.Namespace == testID {
-				return true
-			}
-
-			return false
-		default:
-			return false
-		}
-	}, testTimeout, 1*time.Second).Should(BeTrue())
-	err = listenerHandler.Close(ctx)
-	g.Expect(err).ToNot(HaveOccurred())
-	err = hub.Close(ctx)
-	g.Expect(err).ToNot(HaveOccurred())
-}
-
 func TestAzureDevOpsCommitStatus(t *testing.T) {
 	g := NewWithT(t)
 

tests/integration/azure_test.go

@@ -21,6 +21,7 @@ import (
 	"fmt"
 	"os"
 
+	eventhub "github.com/Azure/azure-event-hubs-go/v3"
 	"github.com/fluxcd/pkg/git"
 	"github.com/fluxcd/test-infra/tftestenv"
 	tfjson "github.com/hashicorp/terraform-json"
@@ -68,24 +69,36 @@ patchesStrategicMerge:
               value: msi
 `
 
-	privateKeyFile, ok := os.LookupEnv("AZUREDEVOPS_SSH")
+	privateKeyFile, ok := os.LookupEnv(envVarGitRepoSSHPath)
 	if !ok {
-		return nil, fmt.Errorf("AZUREDEVOPS_SSH env variable isn't set")
+		return nil, fmt.Errorf("%s env variable isn't set", envVarGitRepoSSHPath)
 	}
 	privateKeyData, err := os.ReadFile(privateKeyFile)
 	if err != nil {
 		return nil, fmt.Errorf("error getting azure devops private key, '%s': %w", privateKeyFile, err)
 	}
 
-	pubKeyFile, ok := os.LookupEnv("AZUREDEVOPS_SSH_PUB")
+	pubKeyFile, ok := os.LookupEnv(envVarGitRepoSSHPubPath)
 	if !ok {
-		return nil, fmt.Errorf("AZUREDEVOPS_SSH_PUB env variable isn't set")
+		return nil, fmt.Errorf("%s env variable isn't set", envVarGitRepoSSHPubPath)
 	}
 	pubKeyData, err := os.ReadFile(pubKeyFile)
 	if err != nil {
 		return nil, fmt.Errorf("error getting ssh pubkey '%s', %w", pubKeyFile, err)
 	}
 
+	c := make(chan []byte, 10)
+	closefn, err := setupEventHubHandler(ctx, c, eventHubSas)
+
+	var notificationCfg = notificationConfig{
+		notificationChan: c,
+		providerType:     "azureeventhub",
+		closeChan:        closefn,
+		secret: map[string]string{
+			"address": eventHubSas,
+		},
+	}
+
 	config := &testConfig{
 		defaultGitTransport: git.HTTP,
 		gitUsername:         git.DefaultPublicKeyAuthUser,
@@ -93,15 +106,15 @@ patchesStrategicMerge:
 		gitPrivateKey:       string(privateKeyData),
 		gitPublicKey:        string(pubKeyData),
 		knownHosts:          azureDevOpsKnownHosts,
-		fleetInfraRepository: repoConfig{
+		fleetInfraRepository: gitUrl{
 			http: fleetInfraRepository["http"].(string),
 			ssh:  fleetInfraRepository["ssh"].(string),
 		},
-		applicationRepository: repoConfig{
+		applicationRepository: gitUrl{
 			http: applicationRepository["http"].(string),
 			ssh:  applicationRepository["ssh"].(string),
 		},
-		notificationURL: eventHubSas,
+		notificationCfg: notificationCfg,
 		sopsArgs:        fmt.Sprintf("--azure-kv %s", sharedSopsId),
 		sopsSecretData: map[string]string{
 			"sops.azure-kv": fmt.Sprintf(`clientId: %s`, outputs["aks_client_id"].Value.(string)),
@@ -116,7 +129,6 @@ patchesStrategicMerge:
 	if err != nil {
 		return nil, err
 	}
-
 	config.defaultAuthOpts = opts
 
 	return config, nil
@@ -134,3 +146,30 @@ func registryLoginACR(ctx context.Context, output map[string]*tfjson.StateOutput
 
 	return registryURL, nil
 }
+
+func setupEventHubHandler(ctx context.Context, c chan []byte, eventHubSas string) (func(), error) {
+	hub, err := eventhub.NewHubFromConnectionString(eventHubSas)
+	if err != nil {
+		return nil, err
+	}
+
+	handler := func(ctx context.Context, event *eventhub.Event) error {
+		c <- event.Data
+		return nil
+	}
+	runtimeInfo, err := hub.GetRuntimeInformation(ctx)
+	if err != nil {
+		return nil, err
+	}
+	listenerHandler, err := hub.Receive(ctx, runtimeInfo.PartitionIDs[0], handler, eventhub.ReceiveWithLatestOffset())
+	if err != nil {
+		return nil, err
+	}
+
+	closefn := func() {
+		listenerHandler.Close(ctx)
+		hub.Close(ctx)
+	}
+
+	return closefn, nil
+}

tests/integration/gcp_test.go

@@ -0,0 +1,178 @@
+/*
+Copyright 2023 The Flux authors
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package integration
+
+import (
+	"context"
+	"fmt"
+	"io"
+	"log"
+	"os"
+	"strings"
+
+	"cloud.google.com/go/pubsub"
+	tfjson "github.com/hashicorp/terraform-json"
+	"google.golang.org/grpc/codes"
+	"google.golang.org/grpc/status"
+
+	"github.com/fluxcd/pkg/git"
+	"github.com/fluxcd/test-infra/tftestenv"
+)
+
+const (
+	gcpSourceRepoKnownHosts = "[source.developers.google.com]:2022 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB5Iy4/cq/gt/fPqe3uyMy4jwv1Alc94yVPxmnwNhBzJqEV5gRPiRk5u4/JJMbbu9QUVAguBABxL7sBZa5PH/xY="
+)
+
+// createKubeConfigGKE constructs kubeconfig for a GKE cluster from the
+// terraform state output at the given kubeconfig path.
+func createKubeConfigGKE(ctx context.Context, state map[string]*tfjson.StateOutput, kcPath string) error {
+	kubeconfigYaml, ok := state["gke_kubeconfig"].Value.(string)
+	if !ok || kubeconfigYaml == "" {
+		return fmt.Errorf("failed to obtain kubeconfig from tf output")
+	}
+	return tftestenv.CreateKubeconfigGKE(ctx, kubeconfigYaml, kcPath)
+}
+
+// registryLoginGCR logs into the Artifact registries using the gcloud
+// and returns a list of test repositories.
+func registryLoginGCR(ctx context.Context, output map[string]*tfjson.StateOutput) (string, error) {
+	project := output["gcp_project_id"].Value.(string)
+	region := output["gcp_region"].Value.(string)
+	repositoryID := output["artifact_registry_id"].Value.(string)
+	artifactRegistryURL, artifactRepoURL := tftestenv.GetGoogleArtifactRegistryAndRepository(project, region, repositoryID)
+	if err := tftestenv.RegistryLoginGCR(ctx, artifactRegistryURL); err != nil {
+		return "", err
+	}
+
+	return artifactRepoURL, nil
+}
+
+func getTestConfigGKE(ctx context.Context, outputs map[string]*tfjson.StateOutput) (*testConfig, error) {
+	sharedSopsId := outputs["sops_id"].Value.(string)
+
+	privateKeyFile, ok := os.LookupEnv(envVarGitRepoSSHPath)
+	if !ok {
+		return nil, fmt.Errorf("%s env variable isn't set", envVarGitRepoSSHPath)
+	}
+	privateKeyData, err := os.ReadFile(privateKeyFile)
+	if err != nil {
+		return nil, fmt.Errorf("error getting gcp source repositories private key, '%s': %w", privateKeyFile, err)
+	}
+
+	pubKeyFile, ok := os.LookupEnv(envVarGitRepoSSHPubPath)
+	if !ok {
+		return nil, fmt.Errorf("%s env variable isn't set", envVarGitRepoSSHPubPath)
+	}
+	pubKeyData, err := os.ReadFile(pubKeyFile)
+	if err != nil {
+		return nil, fmt.Errorf("error getting ssh pubkey '%s', %w", pubKeyFile, err)
+	}
+
+	c := make(chan []byte, 10)
+	projectID := outputs["gcp_project_id"].Value.(string)
+	topicID := outputs["pubsub_topic"].Value.(string)
+
+	fn, err := setupPubSubReceiver(ctx, c, projectID, topicID)
+	if err != nil {
+		return nil, err
+	}
+
+	var notificationCfg = notificationConfig{
+		providerType:     "googlepubsub",
+		providerChannel:  topicID,
+		notificationChan: c,
+		closeChan:        fn,
+		secret: map[string]string{
+			"address": projectID,
+		},
+	}
+
+	config := &testConfig{
+		defaultGitTransport: git.SSH,
+		gitUsername:         "git",
+		gitPrivateKey:       string(privateKeyData),
+		gitPublicKey:        string(pubKeyData),
+		knownHosts:          gcpSourceRepoKnownHosts,
+		fleetInfraRepository: gitUrl{
+			ssh: outputs["fleet_infra_repository"].Value.(string),
+		},
+		applicationRepository: gitUrl{
+			ssh: outputs["application_repository"].Value.(string),
+		},
+		notificationCfg: notificationCfg,
+		sopsArgs:        fmt.Sprintf("--gcp-kms %s", sharedSopsId),
+	}
+
+	opts, err := authOpts(config.fleetInfraRepository.ssh, map[string][]byte{
+		"identity":    []byte(config.gitPrivateKey),
+		"known_hosts": []byte(config.knownHosts),
+	})
+	if err != nil {
+		return nil, err
+	}
+
+	config.defaultAuthOpts = opts
+
+	// In Azure, the repository is initialized with a default branch through
+	// terraform. We have to do it manually here for GCP to prevent errors
+	// when trying to clone later. We only need to do it for the application repository
+	// since flux bootstrap pushes to the main branch.
+	files := make(map[string]io.Reader)
+	files["README.md"] = strings.NewReader("# Flux test repo")
+	tmpDir, err := os.MkdirTemp("", "*-flux-test")
+	if err != nil {
+		return nil, err
+	}
+	defer os.RemoveAll(tmpDir)
+
+	client, err := getRepository(context.Background(), tmpDir, config.applicationRepository.ssh, defaultBranch, config.defaultAuthOpts)
+	if err != nil {
+		return nil, err
+	}
+	err = commitAndPushAll(context.Background(), client, files, defaultBranch)
+	if err != nil {
+		return nil, err
+	}
+
+	return config, nil
+}
+
+func setupPubSubReceiver(ctx context.Context, c chan []byte, projectID string, topicID string) (func(), error) {
+	newCtx, cancel := context.WithCancel(ctx)
+	pubsubClient, err := pubsub.NewClient(newCtx, projectID)
+	if err != nil {
+		cancel()
+		return nil, fmt.Errorf("error creating pubsub client: %s", err)
+	}
+
+	sub := pubsubClient.Subscription(topicID)
+	go func() {
+		err = sub.Receive(ctx, func(ctx context.Context, message *pubsub.Message) {
+			c <- message.Data
+			message.Ack()
+		})
+		if err != nil && status.Code(err) != codes.Canceled {
+			log.Printf("error receiving message in subscription: %s\n", err)
+			return
+		}
+	}()
+
+	return func() {
+		cancel()
+		pubsubClient.Close()
+	}, nil
+}

tests/integration/go.mod

@@ -3,6 +3,7 @@ module github.com/fluxcd/flux2/tests/integration
 go 1.18
 
 require (
+	cloud.google.com/go/pubsub v1.31.0
 	github.com/Azure/azure-event-hubs-go/v3 v3.6.0
 	github.com/fluxcd/helm-controller/api v0.34.1
 	github.com/fluxcd/image-automation-controller/api v0.34.1
@@ -22,6 +23,7 @@ require (
 	github.com/microsoft/azure-devops-go-api/azuredevops v1.0.0-b5
 	github.com/onsi/gomega v1.27.8
 	github.com/whilp/git-urls v1.0.0
+	google.golang.org/grpc v1.55.0
 	k8s.io/api v0.27.3
 	k8s.io/apimachinery v0.27.3
 	k8s.io/client-go v0.27.3
@@ -29,6 +31,10 @@ require (
 )
 
 require (
+	cloud.google.com/go v0.110.2 // indirect
+	cloud.google.com/go/compute v1.19.0 // indirect
+	cloud.google.com/go/compute/metadata v0.2.3 // indirect
+	cloud.google.com/go/iam v1.0.1 // indirect
 	github.com/Azure/azure-amqp-common-go/v4 v4.2.0 // indirect
 	github.com/Azure/azure-sdk-for-go v65.0.0+incompatible // indirect
 	github.com/Azure/go-amqp v1.0.0 // indirect
@@ -74,7 +80,10 @@ require (
 	github.com/google/gnostic v0.6.9 // indirect
 	github.com/google/go-cmp v0.5.9 // indirect
 	github.com/google/gofuzz v1.2.0 // indirect
+	github.com/google/s2a-go v0.1.4 // indirect
 	github.com/google/uuid v1.3.0 // indirect
+	github.com/googleapis/enterprise-certificate-proxy v0.2.3 // indirect
+	github.com/googleapis/gax-go/v2 v2.9.1 // indirect
 	github.com/hashicorp/errwrap v1.0.0 // indirect
 	github.com/hashicorp/go-cleanhttp v0.5.2 // indirect
 	github.com/hashicorp/go-multierror v1.1.1 // indirect
@@ -105,17 +114,20 @@ require (
 	github.com/vbatts/tar-split v0.11.2 // indirect
 	github.com/xanzy/ssh-agent v0.3.3 // indirect
 	github.com/zclconf/go-cty v1.13.0 // indirect
+	go.opencensus.io v0.24.0 // indirect
 	golang.org/x/crypto v0.9.0 // indirect
 	golang.org/x/mod v0.10.0 // indirect
 	golang.org/x/net v0.10.0 // indirect
-	golang.org/x/oauth2 v0.6.0 // indirect
+	golang.org/x/oauth2 v0.8.0 // indirect
 	golang.org/x/sync v0.2.0 // indirect
 	golang.org/x/sys v0.8.0 // indirect
 	golang.org/x/term v0.8.0 // indirect
 	golang.org/x/text v0.9.0 // indirect
 	golang.org/x/time v0.3.0 // indirect
 	golang.org/x/tools v0.9.1 // indirect
+	google.golang.org/api v0.124.0 // indirect
 	google.golang.org/appengine v1.6.7 // indirect
+	google.golang.org/genproto v0.0.0-20230410155749-daa745c078e1 // indirect
 	google.golang.org/protobuf v1.30.0 // indirect
 	gopkg.in/inf.v0 v0.9.1 // indirect
 	gopkg.in/warnings.v0 v0.1.2 // indirect

tests/integration/go.sum

@@ -1,5 +1,16 @@
 cloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=
 cloud.google.com/go v0.34.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=
+cloud.google.com/go v0.110.2 h1:sdFPBr6xG9/wkBbfhmUz/JmZC7X6LavQgcrVINrKiVA=
+cloud.google.com/go v0.110.2/go.mod h1:k04UEeEtb6ZBRTv3dZz4CeJC3jKGxyhl0sAiVVquxiw=
+cloud.google.com/go/compute v1.19.0 h1:+9zda3WGgW1ZSTlVppLCYFIr48Pa35q1uG2N1itbCEQ=
+cloud.google.com/go/compute v1.19.0/go.mod h1:rikpw2y+UMidAe9tISo04EHNOIf42RLYF/q8Bs93scU=
+cloud.google.com/go/compute/metadata v0.2.3 h1:mg4jlk7mCAj6xXp9UJ4fjI9VUI5rubuGBW5aJ7UnBMY=
+cloud.google.com/go/compute/metadata v0.2.3/go.mod h1:VAV5nSsACxMJvgaAuX6Pk2AawlZn8kiOGuCv6gTkwuA=
+cloud.google.com/go/iam v1.0.1 h1:lyeCAU6jpnVNrE9zGQkTl3WgNgK/X+uWwaw0kynZJMU=
+cloud.google.com/go/iam v1.0.1/go.mod h1:yR3tmSL8BcZB4bxByRv2jkSIahVmCtfKZwLYGBalRE8=
+cloud.google.com/go/kms v1.10.2 h1:8UePKEypK3SQ6g+4mn/s/VgE5L7XOh+FwGGRUqvY3Hw=
+cloud.google.com/go/pubsub v1.31.0 h1:aXdyyJz90kA+bor9+6+xHAciMD5mj8v15WqFZ5E0sek=
+cloud.google.com/go/pubsub v1.31.0/go.mod h1:dYmJ3K97NCQ/e4OwZ20rD4Ym3Bu8Gu9m/aJdWQjdcks=
 github.com/AdaLogics/go-fuzz-headers v0.0.0-20230106234847-43070de90fa1 h1:EKPd1INOIyr5hWOWhvpmQpY6tKjeG0hT1s3AMC/9fic=
 github.com/Azure/azure-amqp-common-go/v4 v4.2.0 h1:q/jLx1KJ8xeI8XGfkOWMN9XrXzAfVTkyvCxPvHCjd2I=
 github.com/Azure/azure-amqp-common-go/v4 v4.2.0/go.mod h1:GD3m/WPPma+621UaU6KNjKEo5Hl09z86viKwQjTpV0Q=
@@ -64,6 +75,7 @@ github.com/bwesterb/go-ristretto v1.2.0/go.mod h1:fUIoIZaG73pV5biE2Blr2xEzDoMj7N
 github.com/census-instrumentation/opencensus-proto v0.2.1/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU=
 github.com/cespare/xxhash v1.1.0 h1:a6HrQnmkObjyL+Gs60czilIUGqrzKutQD6XZog3p+ko=
 github.com/cespare/xxhash v1.1.0/go.mod h1:XrSqR1VqqWfGrhpAt58auRo0WTKS1nRRg3ghfAqPWnc=
+github.com/cespare/xxhash/v2 v2.1.1/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
 github.com/cespare/xxhash/v2 v2.2.0 h1:DC2CZ1Ep5Y4k3ZQ899DldepgrayRUGE6BBZ/cd9Cj44=
 github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw=
 github.com/cloudflare/circl v1.1.0/go.mod h1:prBCrKB9DV4poKZY1l9zBXg2QJY7mvgRvtMxxK7fi4I=
@@ -71,7 +83,11 @@ github.com/cloudflare/circl v1.3.3 h1:fE/Qz0QdIGqeWfnwq0RE0R7MI51s0M2E4Ga9kq5AEM
 github.com/cloudflare/circl v1.3.3/go.mod h1:5XYMA4rFBvNIrhs50XuiBJ15vF2pZn4nnUKZrLbUZFA=
 github.com/cncf/udpa/go v0.0.0-20191209042840-269d4d468f6f/go.mod h1:M8M6+tZqaGXZJjfX53e64911xZQV5JYwmTeXPW+k8Sc=
 github.com/cncf/udpa/go v0.0.0-20201120205902-5459f2c99403/go.mod h1:WmhPx2Nbnhtbo57+VJT5O0JRkEi1Wbu0z5j0R8u5Hbk=
+github.com/cncf/udpa/go v0.0.0-20210930031921-04548b0d99d4/go.mod h1:6pvJx4me5XPnfI9Z40ddWsdw2W/uZgQLFXToKeRcDiI=
 github.com/cncf/xds/go v0.0.0-20210312221358-fbca930ec8ed/go.mod h1:eXthEFrGJvWHgFFCl3hGmgk+/aYT6PnTQLykKQRLhEs=
+github.com/cncf/xds/go v0.0.0-20210805033703-aa0b78936158/go.mod h1:eXthEFrGJvWHgFFCl3hGmgk+/aYT6PnTQLykKQRLhEs=
+github.com/cncf/xds/go v0.0.0-20210922020428-25de7278fc84/go.mod h1:eXthEFrGJvWHgFFCl3hGmgk+/aYT6PnTQLykKQRLhEs=
+github.com/cncf/xds/go v0.0.0-20211011173535-cb28da3451f1/go.mod h1:eXthEFrGJvWHgFFCl3hGmgk+/aYT6PnTQLykKQRLhEs=
 github.com/containerd/stargz-snapshotter/estargz v0.14.3 h1:OqlDCK3ZVUO6C3B/5FSkDwbkEETK84kQgEeFwDC+62k=
 github.com/containerd/stargz-snapshotter/estargz v0.14.3/go.mod h1:KY//uOCIkSuNAHhJogcZtrNHdKrA99/FCCRjE3HD36o=
 github.com/cpuguy83/go-md2man/v2 v2.0.0-20190314233015-f79a8a8ca69d/go.mod h1:maD7wRr/U5Z6m/iR4s+kqSMx2CaBsrgA7czyZG/E6dU=
@@ -104,6 +120,7 @@ github.com/envoyproxy/go-control-plane v0.9.1-0.20191026205805-5f8ba28d4473/go.m
 github.com/envoyproxy/go-control-plane v0.9.4/go.mod h1:6rpuAdCZL397s3pYoYcLgu1mIlRU8Am5FuJP05cCM98=
 github.com/envoyproxy/go-control-plane v0.9.9-0.20201210154907-fd9021fe5dad/go.mod h1:cXg6YxExXjJnVBQHBLXeUAgxn2UodCpnH306RInaBQk=
 github.com/envoyproxy/go-control-plane v0.9.9-0.20210512163311-63b5d3c536b0/go.mod h1:hliV/p42l8fGbc6Y9bQ70uLwIvmJyVE5k4iMKlh8wCQ=
+github.com/envoyproxy/go-control-plane v0.9.10-0.20210907150352-cf90f659a021/go.mod h1:AFq3mo9L8Lqqiid3OhADV3RfLJnjiw63cSpi+fDTRC0=
 github.com/envoyproxy/protoc-gen-validate v0.1.0/go.mod h1:iSmxcyjqTsJpI2R4NaDN7+kN2VEUnK/pcBlmesArF7c=
 github.com/evanphx/json-patch/v5 v5.6.0 h1:b91NhWfaz02IuVxO9faSllyAtNXHMPkC5J8sJCLunww=
 github.com/evanphx/json-patch/v5 v5.6.0/go.mod h1:G79N1coSVB93tBe7j6PhzjmR3/2VvlbKOFpnXhI9Bw4=
@@ -180,6 +197,7 @@ github.com/golang-jwt/jwt/v4 v4.2.0/go.mod h1:/xlHOz8bRuivTWchD4jCa+NbatV+wEUSzw
 github.com/golang-jwt/jwt/v4 v4.4.2 h1:rcc4lwaZgFMCZ5jxF9ABolDcIHdBytAFgqFPbSJQAYs=
 github.com/golang-jwt/jwt/v4 v4.4.2/go.mod h1:m21LjoU+eqJr34lmDMbreY2eSTRJ1cv77w39/MY0Ch0=
 github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q=
+github.com/golang/groupcache v0.0.0-20200121045136-8c9f03a8e57e/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
 github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da h1:oI5xCqsCo564l8iNU+DwB5epxmsaqB+rhGL0m5jtYqE=
 github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
 github.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A=
@@ -206,6 +224,7 @@ github.com/google/go-cmp v0.3.0/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMyw
 github.com/google/go-cmp v0.3.1/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=
 github.com/google/go-cmp v0.4.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
 github.com/google/go-cmp v0.5.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
+github.com/google/go-cmp v0.5.3/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
 github.com/google/go-cmp v0.5.5/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
 github.com/google/go-cmp v0.5.9 h1:O2Tfq5qg4qc4AmwVlvv0oLiVAGB7enBSJ2x2DqQFi38=
 github.com/google/go-cmp v0.5.9/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
@@ -215,10 +234,16 @@ github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/
 github.com/google/gofuzz v1.2.0 h1:xRy4A+RhZaiKjJ1bPfwQ8sedCA+YS2YcCHW6ec7JMi0=
 github.com/google/gofuzz v1.2.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
 github.com/google/pprof v0.0.0-20210720184732-4bb14d4b1be1 h1:K6RDEckDVWvDI9JAJYCmNdQXq6neHJOYx3V6jnqNEec=
+github.com/google/s2a-go v0.1.4 h1:1kZ/sQM3srePvKs3tXAvQzo66XfcReoqFpIpIccE7Oc=
+github.com/google/s2a-go v0.1.4/go.mod h1:Ej+mSEMGRnqRzjc7VtF+jdBwYG5fuJfiZ8ELkjEwM0A=
 github.com/google/uuid v1.1.1/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
 github.com/google/uuid v1.1.2/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
 github.com/google/uuid v1.3.0 h1:t6JiXgmwXMjEs8VusXIJk2BXHsn+wx8BZdTaoZ5fu7I=
 github.com/google/uuid v1.3.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
+github.com/googleapis/enterprise-certificate-proxy v0.2.3 h1:yk9/cqRKtT9wXZSsRH9aurXEpJX+U6FLtpYTdC3R06k=
+github.com/googleapis/enterprise-certificate-proxy v0.2.3/go.mod h1:AwSRAtLfXpU5Nm3pW+v7rGDHp09LsPtGY9MduiEsR9k=
+github.com/googleapis/gax-go/v2 v2.9.1 h1:DpTpJqzZ3NvX9zqjhIuI1oVzYZMvboZe+3LoeEIJjHM=
+github.com/googleapis/gax-go/v2 v2.9.1/go.mod h1:4FG3gMrVZlyMp5itSYKMU9z/lBE7+SbnUOvzH2HqbEY=
 github.com/grpc-ecosystem/grpc-gateway v1.16.0/go.mod h1:BDjrQk3hbvj6Nolgz8mAMFbcEtjT1g+wF4CSlocrBnw=
 github.com/hashicorp/errwrap v1.0.0 h1:hLrqtEDnRye3+sgx6z4qVLNuviH3MR5aQ0ykNJa/UYA=
 github.com/hashicorp/errwrap v1.0.0/go.mod h1:YH+1FKiLXxHSkmPseP+kNlulaMuP3n2brvKWEqk/Jc4=
@@ -365,6 +390,8 @@ github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9dec
 github.com/yuin/goldmark v1.4.13/go.mod h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY=
 github.com/zclconf/go-cty v1.13.0 h1:It5dfKTTZHe9aeppbNOda3mN7Ag7sg6QkBNm6TkyFa0=
 github.com/zclconf/go-cty v1.13.0/go.mod h1:YKQzy/7pZ7iq2jNFzy5go57xdxdWoLLpaEp4u238AE0=
+go.opencensus.io v0.24.0 h1:y73uSU6J157QMP2kn2r30vwW1A2W2WFwSCGnAVxeaD0=
+go.opencensus.io v0.24.0/go.mod h1:vNK8G9p7aAivkbmorf4v+7Hgx+Zs0yY+0fOtgBfjQKo=
 go.opentelemetry.io/proto/otlp v0.7.0/go.mod h1:PqfVotwruBrMGOCsRd/89rSnXhoiJIqeYNgFYFoEGnI=
 go.uber.org/atomic v1.10.0 h1:9qC72Qh0+3MqyJbAn8YU5xVq1frD8bn3JtD2oXtafVQ=
 go.uber.org/multierr v1.8.0 h1:dg6GjLku4EH+249NNmoIciG9N/jURbDG+pFlTkhzIC8=
@@ -378,6 +405,7 @@ golang.org/x/crypto v0.0.0-20200820211705-5c72a883971a/go.mod h1:LzIPMQfyMNhhGPh
 golang.org/x/crypto v0.0.0-20210322153248-0c34fe9e7dc2/go.mod h1:T9bdIzuCu7OtxOm1hfPfRQxPLYneinmdGuTeoZ9dtd4=
 golang.org/x/crypto v0.0.0-20210421170649-83a5a9bb288b/go.mod h1:T9bdIzuCu7OtxOm1hfPfRQxPLYneinmdGuTeoZ9dtd4=
 golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
+golang.org/x/crypto v0.0.0-20220314234659-1baeb1ce4c0b/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4=
 golang.org/x/crypto v0.0.0-20220622213112-05595931fe9d/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4=
 golang.org/x/crypto v0.0.0-20220722155217-630584e8d5aa/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4=
 golang.org/x/crypto v0.5.0/go.mod h1:NK/OQwhpMQP3MwtdjgLlYHnH9ebylxKWv3e0fK+mkQU=
@@ -406,6 +434,7 @@ golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLL
 golang.org/x/net v0.0.0-20200226121028-0de0cce0169b/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
 golang.org/x/net v0.0.0-20200822124328-c89045814202/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA=
 golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU=
+golang.org/x/net v0.0.0-20201110031124-69a78807bb2b/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU=
 golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
 golang.org/x/net v0.0.0-20210326060303-6b1517762897/go.mod h1:uSPa2vr4CLtc/ILN5odXGNXS6mhrKVzTaCXzk9m6W3k=
 golang.org/x/net v0.0.0-20210405180319-a5a99cb37ef4/go.mod h1:p54w0d4576C0XHj96bSt6lcn1PtDYWL6XObtHCRCNQM=
@@ -419,8 +448,8 @@ golang.org/x/net v0.10.0 h1:X2//UzNDwYmtCLn7To6G58Wr6f5ahEAQgKNzv9Y951M=
 golang.org/x/net v0.10.0/go.mod h1:0qNGK6F8kojg2nk9dLZ2mShWaEBan6FAoqfSigmmuDg=
 golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
 golang.org/x/oauth2 v0.0.0-20200107190931-bf48bf16ab8d/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
-golang.org/x/oauth2 v0.6.0 h1:Lh8GPgSKBfWSwFvtuWOfeI3aAAnbXTSutYxJiOJFgIw=
+golang.org/x/oauth2 v0.8.0 h1:6dkIjl3j3LtZ/O3sTgZTMsLKSftL/B8Zgq4huOIIUu8=
-golang.org/x/oauth2 v0.6.0/go.mod h1:ycmewcwgD4Rpr3eZJLSB4Kyyljb3qDh40vJ8STE5HKw=
+golang.org/x/oauth2 v0.8.0/go.mod h1:yr7u4HXZRm1R1kBWqr/xKNqewf0plRYoB7sla+BCIXE=
 golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
@@ -473,6 +502,7 @@ golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
 golang.org/x/text v0.3.5/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
 golang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
 golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ=
+golang.org/x/text v0.3.8/go.mod h1:E6s5w1FMmriuDzIBO73fBruAKo1PCIq6d2Q6DHfQ8WQ=
 golang.org/x/text v0.6.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8=
 golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8=
 golang.org/x/text v0.8.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8=
@@ -497,6 +527,8 @@ golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8T
 golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
 golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
 gomodules.xyz/jsonpatch/v2 v2.3.0 h1:8NFhfS6gzxNqjLIYnZxg319wZ5Qjnx4m/CcX+Klzazc=
+google.golang.org/api v0.124.0 h1:dP6Ef1VgOGqQ8eiv4GiY8RhmeyqzovcXBYPDUYG8Syo=
+google.golang.org/api v0.124.0/go.mod h1:xu2HQurE5gi/3t1aFCvhPD781p0a3p11sdunTJ2BlP4=
 google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM=
 google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
 google.golang.org/appengine v1.6.7 h1:FZR1q0exgwxzPzp/aF+VccGrSfxfPpkBqjIIEq3ru6c=
@@ -506,13 +538,19 @@ google.golang.org/genproto v0.0.0-20190819201941-24fa4b261c55/go.mod h1:DMBHOl98
 google.golang.org/genproto v0.0.0-20200513103714-09dca8ec2884/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c=
 google.golang.org/genproto v0.0.0-20200526211855-cb27e3aa2013/go.mod h1:NbSheEEYHJ7i3ixzK3sjbqSGDJWnxyFXZblF3eUsNvo=
 google.golang.org/genproto v0.0.0-20220107163113-42d7afdf6368/go.mod h1:5CzLGKJ67TSI2B9POpiiyGha0AjJvZIUgRMt1dSmuhc=
+google.golang.org/genproto v0.0.0-20230410155749-daa745c078e1 h1:KpwkzHKEF7B9Zxg18WzOa7djJ+Ha5DzthMyZYQfEn2A=
+google.golang.org/genproto v0.0.0-20230410155749-daa745c078e1/go.mod h1:nKE/iIaLqn2bQwXBg8f1g2Ylh6r5MN5CmZvuzZCgsCU=
 google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c=
 google.golang.org/grpc v1.23.0/go.mod h1:Y5yQAOtifL1yxbo5wqy6BxZv8vAUGQwXBOALyacEbxg=
 google.golang.org/grpc v1.25.1/go.mod h1:c3i+UQWmh7LiEpx4sFZnkU36qjEYZ0imhYfXVyQciAY=
 google.golang.org/grpc v1.27.0/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk=
 google.golang.org/grpc v1.33.1/go.mod h1:fr5YgcSWrqhRRxogOsw7RzIpsmvOZ6IcH4kBYTpR3n0=
+google.golang.org/grpc v1.33.2/go.mod h1:JMHMWHQWaTccqQQlmk3MJZS+GWXOdAesneDmEnv2fbc=
 google.golang.org/grpc v1.36.0/go.mod h1:qjiiYl8FncCW8feJPdyg3v6XW24KsRHe+dy9BAGRRjU=
 google.golang.org/grpc v1.40.0/go.mod h1:ogyxbiOoUXAkP+4+xa6PZSE9DZgIHtSpzjDTB9KAK34=
+google.golang.org/grpc v1.45.0/go.mod h1:lN7owxKUQEqMfSyQikvvk5tf/6zMPsrK+ONuO11+0rQ=
+google.golang.org/grpc v1.55.0 h1:3Oj82/tFSCeUrRTg/5E/7d/W5A1tj6Ky1ABAuZuv5ag=
+google.golang.org/grpc v1.55.0/go.mod h1:iYEXKGkEBhg1PjZQvoYEVPTDkHo1/bjTnfwTeGONTY8=
 google.golang.org/protobuf v0.0.0-20200109180630-ec00e32a8dfd/go.mod h1:DFci5gLYBciE7Vtevhsrf46CRTquxDuWsQurQQe4oz8=
 google.golang.org/protobuf v0.0.0-20200221191635-4d8936d0db64/go.mod h1:kwYJMbMJ01Woi6D6+Kah6886xMZcty6N08ah7+eCXa0=
 google.golang.org/protobuf v0.0.0-20200228230310-ab0ca4ff8a60/go.mod h1:cfTl7dwQJ+fmap5saPgwCLgHXTUD7jkjRqWcaiX5VyM=

tests/integration/notification_test.go

@@ -0,0 +1,204 @@
+/*
+Copyright 2023 The Flux authors
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package integration
+
+import (
+	"context"
+	"encoding/json"
+	"io"
+	"strings"
+	"testing"
+	"time"
+
+	. "github.com/onsi/gomega"
+	corev1 "k8s.io/api/core/v1"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/types"
+
+	kustomizev1 "github.com/fluxcd/kustomize-controller/api/v1"
+	notiv1 "github.com/fluxcd/notification-controller/api/v1"
+	notiv1beta2 "github.com/fluxcd/notification-controller/api/v1beta2"
+	events "github.com/fluxcd/pkg/apis/event/v1beta1"
+	"github.com/fluxcd/pkg/apis/meta"
+	sourcev1 "github.com/fluxcd/source-controller/api/v1"
+)
+
+func TestNotification(t *testing.T) {
+	g := NewWithT(t)
+
+	ctx := context.TODO()
+	branchName := "test-notification"
+	testID := branchName + "-" + randStringRunes(5)
+	defer cfg.notificationCfg.closeChan()
+
+	// Setup Flux resources
+	manifest := `apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: foobar`
+	repoUrl := getTransportURL(cfg.applicationRepository)
+	client, err := getRepository(ctx, t.TempDir(), repoUrl, defaultBranch, cfg.defaultAuthOpts)
+	g.Expect(err).ToNot(HaveOccurred())
+	files := make(map[string]io.Reader)
+	files["configmap.yaml"] = strings.NewReader(manifest)
+	err = commitAndPushAll(ctx, client, files, branchName)
+	g.Expect(err).ToNot(HaveOccurred())
+
+	namespace := corev1.Namespace{
+		ObjectMeta: metav1.ObjectMeta{
+			Name: testID,
+		},
+	}
+	g.Expect(testEnv.Create(ctx, &namespace)).To(Succeed())
+	defer testEnv.Delete(ctx, &namespace)
+
+	provider := notiv1beta2.Provider{
+		ObjectMeta: metav1.ObjectMeta{
+			Name:      testID,
+			Namespace: testID,
+		},
+		Spec: notiv1beta2.ProviderSpec{
+			Type:    cfg.notificationCfg.providerType,
+			Address: cfg.notificationCfg.providerAddress,
+			Channel: cfg.notificationCfg.providerChannel,
+		},
+	}
+
+	if cfg.notificationCfg.secret != nil {
+		secret := corev1.Secret{
+			ObjectMeta: metav1.ObjectMeta{
+				Name:      testID,
+				Namespace: testID,
+			},
+			StringData: cfg.notificationCfg.secret,
+		}
+
+		g.Expect(testEnv.Create(ctx, &secret)).To(Succeed())
+		defer testEnv.Delete(ctx, &secret)
+
+		provider.Spec.SecretRef = &meta.LocalObjectReference{
+			Name: testID,
+		}
+	}
+
+	g.Expect(testEnv.Create(ctx, &provider)).To(Succeed())
+	defer testEnv.Delete(ctx, &provider)
+
+	alert := notiv1beta2.Alert{
+		ObjectMeta: metav1.ObjectMeta{
+			Name:      testID,
+			Namespace: testID,
+		},
+		Spec: notiv1beta2.AlertSpec{
+			ProviderRef: meta.LocalObjectReference{
+				Name: provider.Name,
+			},
+			EventSources: []notiv1.CrossNamespaceObjectReference{
+				{
+					Kind:      "Kustomization",
+					Name:      testID,
+					Namespace: testID,
+				},
+			},
+		},
+	}
+	g.Expect(testEnv.Create(ctx, &alert)).ToNot(HaveOccurred())
+	defer testEnv.Delete(ctx, &alert)
+
+	g.Eventually(func() bool {
+		nn := types.NamespacedName{Name: provider.Name, Namespace: provider.Namespace}
+		obj := &notiv1beta2.Provider{}
+		err := testEnv.Get(ctx, nn, obj)
+		if err != nil {
+			return false
+		}
+		if err := checkReadyCondition(obj); err != nil {
+			t.Log(err)
+			return false
+		}
+
+		nn = types.NamespacedName{Name: alert.Name, Namespace: alert.Namespace}
+		alertObj := &notiv1beta2.Alert{}
+		err = testEnv.Get(ctx, nn, alertObj)
+		if err != nil {
+			return false
+		}
+		if err := checkReadyCondition(alertObj); err != nil {
+			t.Log(err)
+			return false
+		}
+
+		return true
+	}, testTimeout, testInterval).Should(BeTrue())
+
+	modifyKsSpec := func(spec *kustomizev1.KustomizationSpec) {
+		spec.Interval = metav1.Duration{Duration: 30 * time.Second}
+		spec.HealthChecks = []meta.NamespacedObjectKindReference{
+			{
+				APIVersion: "v1",
+				Kind:       "ConfigMap",
+				Name:       "foobar",
+				Namespace:  testID,
+			},
+		}
+	}
+	g.Expect(setUpFluxConfig(ctx, testID, nsConfig{
+		repoURL: repoUrl,
+		ref: &sourcev1.GitRepositoryRef{
+			Branch: branchName,
+		},
+		path:         "./",
+		modifyKsSpec: modifyKsSpec,
+	})).To(Succeed())
+	t.Cleanup(func() {
+		err := tearDownFluxConfig(ctx, testID)
+		if err != nil {
+			t.Logf("failed to delete resources in '%s' namespace: %s", testID, err)
+		}
+	})
+
+	g.Eventually(func() bool {
+		err := verifyGitAndKustomization(ctx, testEnv, testID, testID)
+		if err != nil {
+			t.Log(err)
+			return false
+		}
+		return true
+	}, testTimeout, testInterval).Should(BeTrue())
+
+	// Wait to read event from notification channel.
+	g.Eventually(func() bool {
+		select {
+		case eventJson := <-cfg.notificationCfg.notificationChan:
+			event := &events.Event{}
+			err := json.Unmarshal([]byte(eventJson), event)
+			if err != nil {
+				t.Logf("the received event type does not match Flux format, error: %v", err)
+				return false
+			}
+
+			if event.InvolvedObject.Kind == kustomizev1.KustomizationKind &&
+				event.InvolvedObject.Name == testID && event.InvolvedObject.Namespace == testID {
+				return true
+			}
+
+			return false
+		default:
+			return false
+		}
+	}, testTimeout, 1*time.Second).Should(BeTrue())
+}

tests/integration/suite_test.go

@@ -45,17 +45,27 @@ const (
 	// azureTerraformPath is the path to the folder containing the
 	// terraform files for azure infra
 	azureTerraformPath = "./terraform/azure"
+	// gcpTerraformPath is the path to the folder containing the
+	// terraform files for gcp infra
+	gcpTerraformPath = "./terraform/gcp"
 
 	// kubeconfigPath is the path of the file containing the kubeconfig
 	kubeconfigPath = "./build/kubeconfig"
 
 	// default branch to be used when cloning git repositories
 	defaultBranch = "main"
+
+	// envVarGitRepoSSHPath is the environment variable that contains the path
+	// to the ssh key for the git repository
+	envVarGitRepoSSHPath = "GITREPO_SSH_PATH"
+	// envVarGitRepoSSHPubPath is the environment variable that contains the path
+	// to the ssh public key for the git repository
+	envVarGitRepoSSHPubPath = "GITREPO_SSH_PUB_PATH"
 )
 
 var (
 	// supportedProviders are the providers supported by the test.
-	supportedProviders = []string{"azure"}
+	supportedProviders = []string{"azure", "gcp"}
 
 	// cfg is a struct containing different variables needed for the test.
 	cfg *testConfig
@@ -94,18 +104,18 @@ type testConfig struct {
 	defaultGitTransport   git.TransportType
 	defaultAuthOpts       *git.AuthOptions
 	knownHosts            string
-	fleetInfraRepository  repoConfig
+	fleetInfraRepository  gitUrl
-	applicationRepository repoConfig
+	applicationRepository gitUrl
-
-	notificationURL string
 
-	// cloud provider dependent argument to pass to the sops cli
+	// sopsArgs is the cloud provider dependent argument to pass to the sops cli
 	sopsArgs string
-	// secret data for sops
+
+	// notificationCfg contains the values needed to properly set up notification on the
+	// cluster.
+	notificationCfg notificationConfig
+
+	// sopsSecretData is the secret's data for the sops decryption
 	sopsSecretData map[string]string
-	// envCredsData are data field for a secret containing environment variables that the Flux deployments
-	// may need
-	envCredsData map[string]string
 	// kustomizationYaml is the  content of the kustomization.yaml for customizing the Flux manifests
 	kustomizationYaml string
 
@@ -113,9 +123,21 @@ type testConfig struct {
 	testRegistry string
 }
 
-// repoConfig contains the http/ssh urls for the created git repositories
+// notificationConfig contains various fields for configuring
+// providers and testing notifications for the different
+// cloud providers.
+type notificationConfig struct {
+	providerChannel  string
+	providerType     string
+	providerAddress  string
+	secret           map[string]string
+	notificationChan chan []byte
+	closeChan        func()
+}
+
+// gitUrl contains the http/ssh urls for the created git repositories
 // on the various cloud providers.
-type repoConfig struct {
+type gitUrl struct {
 	http string
 	ssh  string
 }
@@ -219,7 +241,7 @@ func TestMain(m *testing.M) {
 	// get provider specific test configuration
 	cfg, err = providerCfg.getTestConfig(ctx, outputs)
 	if err != nil {
-		panic(fmt.Sprintf("Failed to get provider config for %v", err))
+		panic(fmt.Sprintf("Failed to get test config: %v", err))
 	}
 
 	regUrl, err := providerCfg.registryLogin(ctx, outputs)
@@ -269,8 +291,14 @@ func getProviderConfig(provider string) *providerConfig {
 			getTestConfig:    getTestConfigAKS,
 			registryLogin:    registryLoginACR,
 		}
+	case "gcp":
+		return &providerConfig{
+			terraformPath:    gcpTerraformPath,
+			createKubeconfig: createKubeConfigGKE,
+			getTestConfig:    getTestConfigGKE,
+			registryLogin:    registryLoginGCR,
+		}
 	}
-
 	return nil
 }
 

tests/integration/terraform/gcp/gke.tf

@@ -0,0 +1,13 @@
+module "gke" {
+  source = "git::https://github.com/fluxcd/test-infra.git//tf-modules/gcp/gke"
+
+  name = local.name
+  tags = var.tags
+}
+
+module "gcr" {
+  source = "git::https://github.com/fluxcd/test-infra.git//tf-modules/gcp/gcr"
+
+  name = local.name
+  tags = var.tags
+}

tests/integration/terraform/gcp/kms.tf

@@ -0,0 +1,18 @@
+data "google_kms_key_ring" "keyring" {
+  name     = var.gcp_keyring
+  location = "global"
+}
+
+data "google_kms_crypto_key" "my_crypto_key" {
+  name     = var.gcp_crypto_key
+  key_ring = data.google_kms_key_ring.keyring.id
+}
+
+resource "google_kms_key_ring_iam_binding" "key_ring" {
+  key_ring_id = data.google_kms_key_ring.keyring.id
+  role        = "roles/cloudkms.cryptoKeyEncrypterDecrypter"
+
+  members = [
+    "serviceAccount:${data.google_project.project.number}-compute@developer.gserviceaccount.com",
+  ]
+}

tests/integration/terraform/gcp/main.tf

@@ -0,0 +1,13 @@
+provider "google" {
+  project = var.gcp_project_id
+  region  = var.gcp_region
+  zone    = var.gcp_zone
+}
+
+resource "random_pet" "suffix" {}
+
+locals {
+  name = "e2e-${random_pet.suffix.id}"
+}
+
+data "google_project" "project" {}

tests/integration/terraform/gcp/outputs.tf

@@ -0,0 +1,32 @@
+output "gke_kubeconfig" {
+  value     = module.gke.kubeconfig
+  sensitive = true
+}
+
+output "gcp_project_id" {
+  value = var.gcp_project_id
+}
+
+output "gcp_region" {
+  value = var.gcp_region
+}
+
+output "artifact_registry_id" {
+  value = module.gcr.artifact_repository_id
+}
+
+output "sops_id" {
+  value = data.google_kms_crypto_key.my_crypto_key.id
+}
+
+output "fleet_infra_repository" {
+  value = "ssh://${var.gcp_email}@source.developers.google.com:2022/p/${var.gcp_project_id}/r/${google_sourcerepo_repository.fleet-infra.name}"
+}
+
+output "application_repository" {
+  value = "ssh://${var.gcp_email}@source.developers.google.com:2022/p/${var.gcp_project_id}/r/${google_sourcerepo_repository.application.name}"
+}
+
+output "pubsub_topic" {
+  value = google_pubsub_topic.pubsub.name
+}

tests/integration/terraform/gcp/pubsub.tf

@@ -0,0 +1,11 @@
+resource "google_pubsub_topic" "pubsub" {
+  name                       = local.name
+  labels                     = var.tags
+  message_retention_duration = "7200s"
+}
+
+resource "google_pubsub_subscription" "sub" {
+  project = var.gcp_project_id
+  name    = local.name
+  topic   = google_pubsub_topic.pubsub.name
+}

tests/integration/terraform/gcp/sourcerepo.tf

@@ -0,0 +1,7 @@
+resource "google_sourcerepo_repository" "fleet-infra" {
+  name = "fleet-infra-${random_pet.suffix.id}"
+}
+
+resource "google_sourcerepo_repository" "application" {
+  name = "application-${random_pet.suffix.id}"
+}

tests/integration/terraform/gcp/variables.tf

@@ -0,0 +1,37 @@
+variable "gcp_project_id" {
+  type        = string
+  description = "GCP project to create the resources in"
+}
+
+variable "gcp_email" {
+  type        = string
+  description = "GCP user email"
+}
+
+variable "gcp_region" {
+  type        = string
+  default     = "us-central1"
+  description = "GCP region"
+}
+
+variable "gcp_zone" {
+  type        = string
+  default     = "us-central1"
+  description = "GCP zone"
+}
+
+variable "gcp_keyring" {
+  type        = string
+  description = "GCP keyring that contains crypto key for encrypting secrets"
+}
+
+variable "gcp_crypto_key" {
+  type        = string
+  description = "GCP crypto key for encrypting secrets"
+}
+
+variable "tags" {
+  type        = map(string)
+  default     = {}
+  description = "tags for created resources"
+}

tests/integration/util_test.go

@@ -381,12 +381,12 @@ func pushImagesFromURL(repoURL, imgURL string, tags []string) error {
 	return nil
 }
 
-func getTransportURL(repoCfg repoConfig) string {
+func getTransportURL(urls gitUrl) string {
 	if cfg.defaultGitTransport == git.SSH {
-		return repoCfg.ssh
+		return urls.ssh
 	}
 
-	return repoCfg.http
+	return urls.http
 }
 
 func authOpts(repoURL string, authData map[string][]byte) (*git.AuthOptions, error) {