ci: Refactor GitHub workflows

Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>

.github/workflows/README.md

@@ -0,0 +1,50 @@
+# Flux GitHub Workflows
+
+## End-to-end Testing
+
+The e2e workflows run a series of tests to ensure that the Flux CLI and
+the GitOps Toolkit controllers work well all together.
+The tests are written in Go, Bash, Make and Terraform.
+
+| Workflow           | Jobs                 | Runner         | Role                                          |
+|--------------------|----------------------|----------------|-----------------------------------------------|
+| e2e.yaml           | e2e-amd64-kubernetes | GitHub Ubuntu  | integration testing with Kubernetes Kind<br/> |
+| e2e-arm64.yaml     | e2e-arm64-kubernetes | Equinix Ubuntu | integration testing with Kubernetes Kind<br/> |
+| e2e-bootstrap.yaml | e2e-boostrap-github  | GitHub Ubuntu  | integration testing with GitHub API<br/>      |
+| e2e-azure.yaml     | e2e-amd64-aks        | GitHub Ubuntu  | integration testing with Azure API<br/>       |
+| scan.yaml          | scan-fossa           | GitHub Ubuntu  | license scanning<br/>                         |
+| scan.yaml          | scan-snyk            | GitHub Ubuntu  | vulnerability scanning<br/>                   |
+| scan.yaml          | scan-codeql          | GitHub Ubuntu  | vulnerability scanning<br/>                   |
+
+## Components Update
+
+The components update workflow scans the GitOps Toolkit controller repositories for new releases,
+amd when it finds a new controller version, the workflow performs the following steps:
+- Updates the controller API package version in `go.mod`.
+- Patches the controller CRDs version in the `manifests/crds` overlay.
+- Patches the controller Deployment version in `manifests/bases` overlay.
+- Opens a Pull Request against the `main` branch.
+- Triggers the e2e test suite to run for the opened PR.
+
+
+| Workflow    | Jobs              | Runner        | Role                                                |
+|-------------|-------------------|---------------|-----------------------------------------------------|
+| update.yaml | update-components | GitHub Ubuntu | update the GitOps Toolkit APIs and controllers<br/> |
+
+## Release
+
+The release workflow is triggered by a semver Git tag and performs the following steps:
+- Generates the Flux install manifests (YAML).
+- Generates the OpenAPI validation schemas for the GitOps Toolkit CRDs (JSON).
+- Generates a Software Bill of Materials (SPDX JSON).
+- Builds the Flux CLI binaries and the multi-arch container images.
+- Pushes the container images to GitHub Container Registry and DockerHub.
+- Signs the sbom, the binaries checksum and the container images with Cosign and GitHub OIDC.
+- Uploads the sbom, binaries, checksums and install manifests to GitHub Releases.
+- Pushes the install manifests as OCI artifacts to GitHub Container Registry and DockerHub.
+- Signs the OCI artifacts with Cosign and GitHub OIDC.
+
+| Workflow     | Jobs                   | Runner        | Role                                                 |
+|--------------|------------------------|---------------|------------------------------------------------------|
+| release.yaml | release-flux-cli       | GitHub Ubuntu | build, push and sign the CLI release artifacts<br/>  |
+| release.yaml | release-flux-manifests | GitHub Ubuntu | build, push and sign the Flux install manifests<br/> |

.github/workflows/e2e-arm64.yaml

@@ -9,7 +9,7 @@ permissions:
   contents: read
 
 jobs:
-  test:
+  e2e-arm64-kubernetes:
     # Hosted on Equinix
     # Docs: https://github.com/fluxcd/flux2/tree/main/.github/runners
     runs-on: [self-hosted, Linux, ARM64, equinix]

.github/workflows/e2e-azure.yaml

@@ -11,7 +11,7 @@ permissions:
   contents: read
 
 jobs:
-  e2e:
+  e2e-amd64-aks:
     runs-on: ubuntu-22.04
     steps:
       - name: Checkout

.github/workflows/e2e-bootstrap.yaml

@@ -1,6 +1,7 @@
-name: bootstrap
+name: e2e-bootstrap
 
 on:
+  workflow_dispatch:
   push:
     branches: [ main ]
   pull_request:
@@ -10,7 +11,7 @@ permissions:
   contents: read
 
 jobs:
-  github:
+  e2e-boostrap-github:
     runs-on: ubuntu-latest
     if: (github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository) && github.actor != 'dependabot[bot]'
     steps:

.github/workflows/e2e.yaml

@@ -1,6 +1,7 @@
 name: e2e
 
 on:
+  workflow_dispatch:
   push:
     branches: [ main ]
   pull_request:
@@ -10,7 +11,7 @@ permissions:
   contents: read
 
 jobs:
-  kind:
+  e2e-amd64-kubernetes:
     runs-on: ubuntu-latest
     services:
       registry:

.github/workflows/release-manifests.yml

@@ -1,75 +0,0 @@
-name: release-manifests
-on:
-  release:
-    types: [published]
-  workflow_dispatch:
-
-permissions:
-  contents: read
-
-jobs:
-  build-push:
-    runs-on: ubuntu-latest
-    permissions:
-      id-token: write # needed for keyless signing
-      packages: write # needed for ghcr access
-    steps:
-      - uses: actions/checkout@v3
-      - name: Setup Kustomize
-        uses: fluxcd/pkg/actions/kustomize@main
-      - name: Setup Flux CLI
-        uses: ./action/
-      - name: Prepare
-        id: prep
-        run: |
-          VERSION=$(flux version --client | awk '{ print $NF }')
-          echo ::set-output name=VERSION::${VERSION}
-      - name: Login to GHCR
-        uses: docker/login-action@v2
-        with:
-          registry: ghcr.io
-          username: fluxcdbot
-          password: ${{ secrets.GHCR_TOKEN }}
-      - name: Login to DockerHub
-        uses: docker/login-action@v2
-        with:
-          username: fluxcdbot
-          password: ${{ secrets.DOCKER_FLUXCD_PASSWORD }}
-      - name: Push manifests to GHCR
-        run: |
-          mkdir -p ./ghcr.io/flux-system
-          flux install --registry=ghcr.io/fluxcd \
-          --components-extra=image-reflector-controller,image-automation-controller \
-          --export > ./ghcr.io/flux-system/gotk-components.yaml
-          
-          cd ./ghcr.io && flux push artifact \
-          oci://ghcr.io/fluxcd/flux-manifests:${{ steps.prep.outputs.VERSION }} \
-          --path="./flux-system" \
-          --source=${{ github.repositoryUrl }} \
-          --revision="${{ github.ref_name }}/${{ github.sha }}"
-      - name: Push manifests to DockerHub
-        run: |
-          mkdir -p ./docker.io/flux-system
-          flux install --registry=docker.io/fluxcd \
-          --components-extra=image-reflector-controller,image-automation-controller \
-          --export > ./docker.io/flux-system/gotk-components.yaml
-          
-          cd ./docker.io && flux push artifact \
-          oci://docker.io/fluxcd/flux-manifests:${{ steps.prep.outputs.VERSION }} \
-          --path="./flux-system" \
-          --source=${{ github.repositoryUrl }} \
-          --revision="${{ github.ref_name }}/${{ github.sha }}"
-      - uses: sigstore/cosign-installer@main
-      - name: Sign manifests
-        env:
-          COSIGN_EXPERIMENTAL: 1
-        run: |
-          cosign sign ghcr.io/fluxcd/flux-manifests:${{ steps.prep.outputs.VERSION }}
-          cosign sign docker.io/fluxcd/flux-manifests:${{ steps.prep.outputs.VERSION }}
-      - name: Tag manifests
-        run: |
-          flux tag artifact oci://ghcr.io/fluxcd/flux-manifests:${{ steps.prep.outputs.VERSION }} \
-          --tag latest
-
-          flux tag artifact oci://docker.io/fluxcd/flux-manifests:${{ steps.prep.outputs.VERSION }} \
-          --tag latest

.github/workflows/release.yaml

@@ -8,12 +8,12 @@ permissions:
   contents: read
 
 jobs:
-  goreleaser:
+  release-flux-cli:
-    permissions: # TODO: Segment these jobs to minimize which actions are recieving escalated perms
+    runs-on: ubuntu-latest
+    permissions:
       contents: write # needed to write releases
       id-token: write # needed for keyless signing
       packages: write # needed for ghcr access
-    runs-on: ubuntu-latest
     steps:
       - name: Checkout
         uses: actions/checkout@v3
@@ -83,3 +83,69 @@ jobs:
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
           HOMEBREW_TAP_GITHUB_TOKEN: ${{ secrets.BOT_GITHUB_TOKEN }}
           AUR_BOT_SSH_PRIVATE_KEY: ${{ secrets.AUR_BOT_SSH_PRIVATE_KEY }}
+  release-flux-manifests:
+    runs-on: ubuntu-latest
+    needs: release-flux-cli
+    permissions:
+      id-token: write
+      packages: write
+    steps:
+      - uses: actions/checkout@v3
+      - name: Setup Kustomize
+        uses: fluxcd/pkg/actions/kustomize@main
+      - name: Setup Flux CLI
+        uses: ./action/
+      - name: Prepare
+        id: prep
+        run: |
+          VERSION=$(flux version --client | awk '{ print $NF }')
+          echo ::set-output name=VERSION::${VERSION}
+      - name: Login to GHCR
+        uses: docker/login-action@v2
+        with:
+          registry: ghcr.io
+          username: fluxcdbot
+          password: ${{ secrets.GHCR_TOKEN }}
+      - name: Login to DockerHub
+        uses: docker/login-action@v2
+        with:
+          username: fluxcdbot
+          password: ${{ secrets.DOCKER_FLUXCD_PASSWORD }}
+      - name: Push manifests to GHCR
+        run: |
+          mkdir -p ./ghcr.io/flux-system
+          flux install --registry=ghcr.io/fluxcd \
+          --components-extra=image-reflector-controller,image-automation-controller \
+          --export > ./ghcr.io/flux-system/gotk-components.yaml
+          
+          cd ./ghcr.io && flux push artifact \
+          oci://ghcr.io/fluxcd/flux-manifests:${{ steps.prep.outputs.VERSION }} \
+          --path="./flux-system" \
+          --source=${{ github.repositoryUrl }} \
+          --revision="${{ github.ref_name }}/${{ github.sha }}"
+      - name: Push manifests to DockerHub
+        run: |
+          mkdir -p ./docker.io/flux-system
+          flux install --registry=docker.io/fluxcd \
+          --components-extra=image-reflector-controller,image-automation-controller \
+          --export > ./docker.io/flux-system/gotk-components.yaml
+          
+          cd ./docker.io && flux push artifact \
+          oci://docker.io/fluxcd/flux-manifests:${{ steps.prep.outputs.VERSION }} \
+          --path="./flux-system" \
+          --source=${{ github.repositoryUrl }} \
+          --revision="${{ github.ref_name }}/${{ github.sha }}"
+      - uses: sigstore/cosign-installer@main
+      - name: Sign manifests
+        env:
+          COSIGN_EXPERIMENTAL: 1
+        run: |
+          cosign sign ghcr.io/fluxcd/flux-manifests:${{ steps.prep.outputs.VERSION }}
+          cosign sign docker.io/fluxcd/flux-manifests:${{ steps.prep.outputs.VERSION }}
+      - name: Tag manifests
+        run: |
+          flux tag artifact oci://ghcr.io/fluxcd/flux-manifests:${{ steps.prep.outputs.VERSION }} \
+          --tag latest
+
+          flux tag artifact oci://docker.io/fluxcd/flux-manifests:${{ steps.prep.outputs.VERSION }} \
+          --tag latest

.github/workflows/scan.yaml

@@ -1,6 +1,7 @@
 name: scan
 
 on:
+  workflow_dispatch:
   push:
     branches: [ main ]
   pull_request:
@@ -9,11 +10,10 @@ on:
     - cron: '18 10 * * 3'
 
 permissions:
-  contents: read # for actions/checkout to fetch code
+  contents: read
 
 jobs:
-  fossa:
+  scan-fossa:
-    name: FOSSA
     runs-on: ubuntu-latest
     if: github.actor != 'dependabot[bot]'
     steps:
@@ -25,11 +25,10 @@ jobs:
           fossa-api-key: 5ee8bf422db1471e0bcf2bcb289185de
           github-token: ${{ github.token }}
 
-  snyk:
+  scan-snyk:
-    name: Snyk
-    permisions:
-      security-events: write
     runs-on: ubuntu-latest
+    permissions:
+      security-events: write
     if: (github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository) && github.actor != 'dependabot[bot]'
     steps:
       - uses: actions/checkout@v3
@@ -50,11 +49,10 @@ jobs:
         with:
           sarif_file: snyk.sarif
 
-  codeql:
+  scan-codeql:
-    name: CodeQL
     runs-on: ubuntu-latest
     permissions:
-      security-events: write # for codeQL to write security events
+      security-events: write
     if: github.actor != 'dependabot[bot]'
     steps:
       - name: Checkout repository

.github/workflows/update.yaml

@@ -1,4 +1,4 @@
-name: Update Components
+name: update
 
 on:
   workflow_dispatch:
@@ -12,10 +12,10 @@ permissions:
 
 jobs:
   update-components:
+    runs-on: ubuntu-latest
     permissions:
       contents: write
       pull-requests: write
-    runs-on: ubuntu-latest
     steps:
       - name: Check out code
         uses: actions/checkout@v3