monitoring: Configure KSM & cluster dashboard

Update kube-prometheus-stack helm release values to configure
kube-state-metrics and use kube-state-metrics to collect gotk resource
state metrics.

- Configure kube-state-metrics to run in custom resource state only
  mode. In this mode, it'll only watch custom resources. Also, pass
  empty collectors as extra args to prevent passing all the core
  resources to watch as an argument.
- Running kube-state-metrics in custom resource state only mode makes
  the default grafana dashboards of no use. Disable the default
  dashboards.
- Add kube-state-metrics configuration to provide RBAC permissions to it
  to allow listing and watching flux CRDs.
- Also, configure custom resource state for each of the flux custom
  resources using Info type metrics called `gotk_resource_info`. KSM
  issues a warning if an Info type object doesn't have `_info` suffix.
  These metrics have the value 1 always. This works well for the CRD
  state metrics as a zero value would mean that the resource doesn't
  exist, in which case, the resource is deleted.
- Update the cluster dashboard panels to use `gotk_resource_info` in the
  queries.
  - Only the following panels have been updated
    - Cluster Reconcilers
    - Failing Reconcilers
    - Cluster reconciliation readiness
    - Kubernetes Manifests Sources
    - Failing Sources
    - Source acquisition readiness
  - The panels have been updated such that it's work with static
  resources which don't have any status as well. By default, it assumes
  such static resources to be in a Ready state. Resources are seen as
  failed only when the ready value is false.
  - The queries have been updated to Instant type in order to show the
  current data, instead of the result of past 15 minutes. This shows
  more accurate resource data as the resource metrics change.
  - The Stat visualizers have been updated to have zero as the default
  value when there's no data. This is to prevent showing no data when
  there's no object. This was motivated by the behavior of the previous
  configuration which depended on stale metrics from controllers and
  deleted conditions to show zero value when objects get deleted. With
  the fixes in the controller metrics that removes stale metrics, this
  will no longer work. In order to show a zero value for these stats, a
  default is set.
  - The `$namespace` variable has been updated to refer to
  `exported_namespace` from `gotk_resource_info`.

Signed-off-by: Sunny <darkowlzz@protonmail.com>

.github/workflows/action.yaml

@@ -0,0 +1,29 @@
+name: test-gh-action
+
+on:
+  pull_request:
+    paths:
+      - 'action/**'
+  push:
+    paths:
+      - 'action/**'
+    branches:
+      - 'main'
+      - 'release/**'
+
+permissions: read-all
+
+jobs:
+  actions:
+    strategy:
+      fail-fast: false
+      matrix:
+        version: [ubuntu-latest, macos-latest, windows-latest]
+
+    runs-on: ${{ matrix.version }}
+    name: action on ${{ matrix.version }}
+    steps:
+      - name: Checkout
+        uses: actions/checkout@c85c95e3d7251135ab7dc9ce3241c5835cc595a9 # v3.5.3
+      - name: Setup flux
+        uses: ./action

action/README.md

@@ -1,216 +1,22 @@
 # Flux GitHub Action
 
-Usage:
+To install the latest Flux CLI on Linux, macOS or Windows GitHub runners:
 
 ```yaml
-    steps:
+steps:
-      - name: Setup Flux CLI
+  - name: Setup Flux CLI
-        uses: fluxcd/flux2/action@main
+    uses: fluxcd/flux2/action@main
-      - name: Run Flux commands
+    with:
-        run: flux -v
+      version: 'latest'
+  - name: Run Flux CLI
+    run: flux version --client
 ```
 
-The latest stable version of the `flux` binary is downloaded from
+The Flux GitHub Action can be used to automate various tasks in CI, such as:
-GitHub [releases](https://github.com/fluxcd/flux2/releases)
-and placed at `/usr/local/bin/flux`.
 
-Note that this action can only be used on GitHub **Linux** runners.
+- [Automate Flux upgrades on clusters via Pull Requests](https://fluxcd.io/flux/flux-gh-action/#automate-flux-updates)
-You can change the arch (defaults to `amd64`) with:
+- [Push Kubernetes manifests to container registries](https://fluxcd.io/flux/flux-gh-action/#push-kubernetes-manifests-to-container-registries)
+- [Run end-to-end testing with Flux and Kubernetes Kind](https://fluxcd.io/flux/flux-gh-action/#end-to-end-testing)
 
-```yaml
+For more information, please see the [Flux GitHub Action documentation](/flux/flux-gh-action.md).
-    steps:
-      - name: Setup Flux CLI
-        uses: fluxcd/flux2/action@main
-        with:
-          arch: arm64 # can be amd64, arm64 or arm
-```
 
-You can download a specific version with:
-
-```yaml
-    steps:
-      - name: Setup Flux CLI
-        uses: fluxcd/flux2/action@main
-        with:
-          version: 0.32.0
-```
-
-You can also authenticate against the GitHub API using GitHub Actions' `GITHUB_TOKEN` secret.
-
-For more information, please [read about the GitHub token secret](https://docs.github.com/en/actions/security-guides/automatic-token-authentication#about-the-github_token-secret).
-
-```yaml
-    steps:
-      - name: Setup Flux CLI
-        uses: fluxcd/flux2/action@main
-        with:
-          token: ${{ secrets.GITHUB_TOKEN }}
-```
-
-This is useful if you are seeing failures on shared runners, those failures are usually API limits being hit.
-
-### Automate Flux updates
-
-Example workflow for updating Flux's components generated with `flux bootstrap --path=clusters/production`:
-
-```yaml
-name: update-flux
-
-on:
-  workflow_dispatch:
-  schedule:
-    - cron: "0 * * * *"
-
-permissions:
-  contents: write
-  pull-requests: write
-
-jobs:
-  components:
-    runs-on: ubuntu-latest
-    steps:
-      - name: Check out code
-        uses: actions/checkout@v3
-      - name: Setup Flux CLI
-        uses: fluxcd/flux2/action@main
-      - name: Check for updates
-        id: update
-        run: |
-          flux install \
-            --export > ./clusters/production/flux-system/gotk-components.yaml
-
-          VERSION="$(flux -v)"
-          echo "flux_version=$VERSION" >> $GITHUB_OUTPUT
-      - name: Create Pull Request
-        uses: peter-evans/create-pull-request@v4
-        with:
-            token: ${{ secrets.GITHUB_TOKEN }}
-            branch: update-flux
-            commit-message: Update to ${{ steps.update.outputs.flux_version }}
-            title: Update to ${{ steps.update.outputs.flux_version }}
-            body: |
-              ${{ steps.update.outputs.flux_version }}
-```
-
-### Push Kubernetes manifests to container registries
-
-Example workflow for publishing Kubernetes manifests bundled as OCI artifacts to GitHub Container Registry:
-
-```yaml
-name: push-artifact-staging
-
-on:
-  push:
-    branches:
-      - 'main'
-
-permissions:
-  packages: write # needed for ghcr.io access
-
-env:
-  OCI_REPO: "oci://ghcr.io/my-org/manifests/${{ github.event.repository.name }}"
-
-jobs:
-  kubernetes:
-    runs-on: ubuntu-latest
-    steps:
-      - name: Checkout
-        uses: actions/checkout@v3
-      - name: Setup Flux CLI
-        uses: fluxcd/flux2/action@main
-      - name: Login to GHCR
-        uses: docker/login-action@v2
-        with:
-          registry: ghcr.io
-          username: ${{ github.actor }}
-          password: ${{ secrets.GITHUB_TOKEN }}
-      - name: Generate manifests
-        run: |
-          kustomize build ./manifests/staging > ./deploy/app.yaml
-      - name: Push manifests
-        run: |
-          flux push artifact $OCI_REPO:$(git rev-parse --short HEAD) \
-            --path="./deploy" \
-            --source="$(git config --get remote.origin.url)" \
-            --revision="$(git branch --show-current)@sha1:$(git rev-parse HEAD)"
-      - name: Deploy manifests to staging
-        run: |
-          flux tag artifact $OCI_REPO:$(git rev-parse --short HEAD) --tag staging
-```
-
-### Push and sign Kubernetes manifests to container registries
-
-Example workflow for publishing Kubernetes manifests bundled as OCI artifacts
-which are signed with Cosign and GitHub OIDC:
-
-```yaml
-name: push-sign-artifact
-
-on:
-  push:
-    branches:
-      - 'main'
-
-permissions:
-  packages: write # needed for ghcr.io access
-  id-token: write # needed for keyless signing
-
-env:
-  OCI_REPO: "oci://ghcr.io/my-org/manifests/${{ github.event.repository.name }}"
-
-jobs:
-  kubernetes:
-    runs-on: ubuntu-latest
-    steps:
-      - name: Checkout
-        uses: actions/checkout@v3
-      - name: Setup Flux CLI
-        uses: fluxcd/flux2/action@main
-      - name: Setup Cosign
-        uses: sigstore/cosign-installer@main
-      - name: Login to GHCR
-        uses: docker/login-action@v2
-        with:
-          registry: ghcr.io
-          username: ${{ github.actor }}
-          password: ${{ secrets.GITHUB_TOKEN }}
-      - name: Push and sign manifests
-        run: |
-          digest_url=$(flux push artifact \
-          $OCI_REPO:$(git rev-parse --short HEAD) \
-          --path="./manifests" \
-          --source="$(git config --get remote.origin.url)" \
-          --revision="$(git branch --show-current)@sha1:$(git rev-parse HEAD)" |\
-          jq -r '. | .repository + "@" + .digest')
-
-          cosign sign $digest_url
-```
-
-### End-to-end testing
-
-Example workflow for running Flux in Kubernetes Kind:
-
-```yaml
-name: e2e
-
-on:
-  push:
-    branches:
-      - '*'
-
-jobs:
-  kubernetes:
-    runs-on: ubuntu-latest
-    steps:
-      - name: Checkout
-        uses: actions/checkout@v3
-      - name: Setup Flux CLI
-        uses: fluxcd/flux2/action@main
-      - name: Setup Kubernetes Kind
-        uses: engineerd/setup-kind@v0.5.0
-      - name: Install Flux in Kubernetes Kind
-        run: flux install
-```
-
-A complete e2e testing workflow is available here
-[flux2-kustomize-helm-example](https://github.com/fluxcd/flux2-kustomize-helm-example/blob/main/.github/workflows/e2e.yaml)

action/action.yml

@@ -1,64 +1,120 @@
 name: Setup Flux CLI
-description: A GitHub Action for running Flux commands
+description: A GitHub Action for installing the Flux CLI
-author: Stefan Prodan
+author: Flux project
 branding:
   color: blue
   icon: command
 inputs:
   version:
-    description: "Flux version e.g. 0.8.0 (defaults to latest stable release)"
+    description: "Flux version e.g. 2.0.0 (defaults to latest stable release)"
     required: false
   arch:
     description: "arch can be amd64, arm64 or arm"
-    required: true
+    required: false
-    default: "amd64"
+    deprecationMessage: "No longer required, action will now detect runner arch."
   bindir:
-    description: "Optional location of the Flux binary. Will not use sudo if set. Updates System Path."
+    description: "Alternative location for the Flux binary, defaults to path relative to $RUNNER_TOOL_CACHE."
     required: false
   token:
-    description: "GitHub Token used to authentication against the API (generally only needed to prevent quota limit errors)"
+    description: "Token used to authentication against the GitHub.com API. Defaults to the token from the GitHub context of the workflow."
     required: false
 runs:
   using: composite
   steps:
-    - name: "Download flux binary to tmp"
+    - name: "Download the binary to the runner's cache dir"
       shell: bash
       run: |
-        ARCH=${{ inputs.arch }}
         VERSION=${{ inputs.version }}
-        TOKEN=${{ inputs.token }}
 
-        if [ -z "${VERSION}" ]; then
+        TOKEN=${{ inputs.token }}
-          if [ -n "${TOKEN}" ]; then
+        if [[ -z "$TOKEN" ]]; then
-            VERSION_SLUG=$(curl https://api.github.com/repos/fluxcd/flux2/releases/latest --silent --location --header "Authorization: token ${TOKEN}" | grep tag_name)
+          TOKEN=${{ github.token }}
-          else
+        fi
-            # With no GITHUB_TOKEN you will experience occasional failures due to rate limiting
+
-            # Ref: https://github.com/fluxcd/flux2/issues/3509#issuecomment-1400820992
+        if [[ -z "$VERSION" ]] || [[ "$VERSION" = "latest" ]]; then
-            VERSION_SLUG=$(curl https://api.github.com/repos/fluxcd/flux2/releases/latest --silent --location | grep tag_name)
+          VERSION=$(curl -fsSL -H "Authorization: token ${TOKEN}" https://api.github.com/repos/fluxcd/flux2/releases/latest | grep tag_name | cut -d '"' -f 4)
+        fi
+        if [[ -z "$VERSION" ]]; then
+          echo "Unable to determine Flux CLI version"
+          exit 1
+        fi
+        if [[ $VERSION = v* ]]; then
+          VERSION="${VERSION:1}"
+        fi
+
+        OS=$(echo "${RUNNER_OS}" | tr '[:upper:]' '[:lower:]')
+        if [[ "$OS" == "macos" ]]; then
+          OS="darwin"
+        fi
+
+        ARCH=$(echo "${RUNNER_ARCH}" | tr '[:upper:]' '[:lower:]')
+        if [[ "$ARCH" == "x64" ]]; then
+          ARCH="amd64"
+        elif [[ "$ARCH" == "x86" ]]; then
+          ARCH="386"
+        fi
+
+        FLUX_EXEC_FILE="flux"
+        if [[ "$OS" == "windows" ]]; then
+            FLUX_EXEC_FILE="${FLUX_EXEC_FILE}.exe"
+        fi
+
+        FLUX_TOOL_DIR=${{ inputs.bindir }}
+        if [[ -z "$FLUX_TOOL_DIR" ]]; then
+          FLUX_TOOL_DIR="${RUNNER_TOOL_CACHE}/flux2/${VERSION}/${OS}/${ARCH}"
+        fi
+        if [[ ! -x "$FLUX_TOOL_DIR/FLUX_EXEC_FILE" ]]; then
+          DL_DIR="$(mktemp -dt flux2-XXXXXX)"
+          trap 'rm -rf $DL_DIR' EXIT
+
+          echo "Downloading flux ${VERSION} for ${OS}/${ARCH}"
+          FLUX_TARGET_FILE="flux_${VERSION}_${OS}_${ARCH}.tar.gz"
+          if [[ "$OS" == "windows" ]]; then
+            FLUX_TARGET_FILE="flux_${VERSION}_${OS}_${ARCH}.zip"
           fi
 
-          VERSION=$(echo "${VERSION_SLUG}" | sed -E 's/.*"([^"]+)".*/\1/' | cut -c 2-)
+          FLUX_CHECKSUMS_FILE="flux_${VERSION}_checksums.txt"
+
+          FLUX_DOWNLOAD_URL="https://github.com/fluxcd/flux2/releases/download/v${VERSION}/"
+
+          curl -fsSL -o "$DL_DIR/$FLUX_TARGET_FILE" "$FLUX_DOWNLOAD_URL/$FLUX_TARGET_FILE"
+          curl -fsSL -o "$DL_DIR/$FLUX_CHECKSUMS_FILE" "$FLUX_DOWNLOAD_URL/$FLUX_CHECKSUMS_FILE"
+
+          echo "Verifying checksum"
+          sum=""
+          if command -v openssl > /dev/null; then
+            sum=$(openssl sha256 "$DL_DIR/$FLUX_TARGET_FILE" | awk '{print $2}')
+          elif command -v sha256sum > /dev/null; then
+            sum=$(sha256sum "$DL_DIR/$FLUX_TARGET_FILE" | awk '{print $1}')
+          fi
+
+          if [[ -z "$sum" ]]; then
+            echo "Neither openssl nor sha256sum found. Cannot calculate checksum."
+            exit 1
+          fi
+
+          expected_sum=$(grep " $FLUX_TARGET_FILE\$" "$DL_DIR/$FLUX_CHECKSUMS_FILE" | awk '{print $1}')
+          if [ "$sum" != "$expected_sum" ]; then
+            echo "SHA sum of ${FLUX_TARGET_FILE} does not match. Aborting."
+            exit 1
+          fi
+
+          echo "Installing flux to ${FLUX_TOOL_DIR}"
+          mkdir -p "$FLUX_TOOL_DIR"
+        
+          if [[ "$OS" == "windows" ]]; then
+            unzip "$DL_DIR/$FLUX_TARGET_FILE" "$FLUX_EXEC_FILE" -d "$FLUX_TOOL_DIR"
+          else
+            tar xzf "$DL_DIR/$FLUX_TARGET_FILE" -C "$FLUX_TOOL_DIR" $FLUX_EXEC_FILE
+          fi
+
+          chmod +x "$FLUX_TOOL_DIR/$FLUX_EXEC_FILE"
         fi
 
-        BIN_URL="https://github.com/fluxcd/flux2/releases/download/v${VERSION}/flux_${VERSION}_linux_${ARCH}.tar.gz"
+        echo "Adding flux to path"
-        curl --silent --fail --location "${BIN_URL}" --output /tmp/flux.tar.gz
+        echo "$FLUX_TOOL_DIR" >> "$GITHUB_PATH"
-        mkdir -p /tmp/flux
+
-        tar -C /tmp/flux/ -zxvf /tmp/flux.tar.gz
+    - name: "Print installed flux version"
-    - name: "Copy Flux binary to execute location"
-      shell: bash
-      run: |
-        BINDIR=${{ inputs.bindir }}
-        if [ -z "${BINDIR}" ]; then
-          sudo cp /tmp/flux/flux /usr/local/bin
-        else
-          cp /tmp/flux/flux "${BINDIR}"
-          echo "${BINDIR}" >> $GITHUB_PATH
-        fi
-    - name: "Cleanup tmp"
-      shell: bash
-      run: |
-        rm -rf /tmp/flux/ /tmp/flux.tar.gz
-    - name: "Verify correct installation of binary"
       shell: bash
       run: |
         flux -v

manifests/monitoring/kube-prometheus-stack/release.yaml

@@ -6,7 +6,7 @@ spec:
   interval: 5m
   chart:
     spec:
-      version: "45.x"
+      version: "48.x"
       chart: kube-prometheus-stack
       sourceRef:
         kind: HelmRepository
@@ -31,6 +31,249 @@ spec:
         podMonitorSelector:
           matchLabels:
             app.kubernetes.io/component: monitoring
+    grafana:
+      defaultDashboardsEnabled: false
+    kube-state-metrics:
+      collectors: []
+      extraArgs:
+        - --custom-resource-state-only=true
+      rbac:
+        extraRules:
+          - apiGroups:
+              - source.toolkit.fluxcd.io
+              - kustomize.toolkit.fluxcd.io
+              - helm.toolkit.fluxcd.io
+              - image.toolkit.fluxcd.io
+              - notification.toolkit.fluxcd.io
+            resources:
+              - gitrepositories
+              - buckets
+              - helmrepositories
+              - helmcharts
+              - ocirepositories
+              - kustomizations
+              - helmreleases
+              - imagerepositories
+              - imagepolicies
+              - imageupdateautomations
+              - alerts
+              - providers
+              - receivers
+            verbs: ["list", "watch"]
+      customResourceState:
+        enabled: true
+        config:
+          spec:
+            resources:
+              - groupVersionKind:
+                  group: source.toolkit.fluxcd.io
+                  version: "v1"
+                  kind: GitRepository
+                metricNamePrefix: gotk
+                metrics:
+                  - name: "resource_info"
+                    help: "The current state of a GitOps Toolkit resource."
+                    each:
+                      type: Info
+                      info:
+                        labelsFromPath:
+                          name: [metadata, name]
+                    labelsFromPath:
+                      exported_namespace: [metadata, namespace]
+                      ready: [status, conditions, "[type=Ready]", status]
+              - groupVersionKind:
+                  group: source.toolkit.fluxcd.io
+                  version: "v1beta2"
+                  kind: Bucket
+                metricNamePrefix: gotk
+                metrics:
+                  - name: "resource_info"
+                    help: "The current state of a GitOps Toolkit resource."
+                    each:
+                      type: Info
+                      info:
+                        labelsFromPath:
+                          name: [metadata, name]
+                    labelsFromPath:
+                      exported_namespace: [metadata, namespace]
+                      ready: [status, conditions, "[type=Ready]", status]
+              - groupVersionKind:
+                  group: source.toolkit.fluxcd.io
+                  version: "v1beta2"
+                  kind: HelmRepository
+                metricNamePrefix: gotk
+                metrics:
+                  - name: "resource_info"
+                    help: "The current state of a GitOps Toolkit resource."
+                    each:
+                      type: Info
+                      info:
+                        labelsFromPath:
+                          name: [metadata, name]
+                    labelsFromPath:
+                      exported_namespace: [metadata, namespace]
+                      type: [spec, type]
+                      ready: [status, conditions, "[type=Ready]", status]
+              - groupVersionKind:
+                  group: source.toolkit.fluxcd.io
+                  version: "v1beta2"
+                  kind: HelmChart
+                metricNamePrefix: gotk
+                metrics:
+                  - name: "resource_info"
+                    help: "The current state of a GitOps Toolkit resource."
+                    each:
+                      type: Info
+                      info:
+                        labelsFromPath:
+                          name: [metadata, name]
+                    labelsFromPath:
+                      exported_namespace: [metadata, namespace]
+                      ready: [status, conditions, "[type=Ready]", status]
+              - groupVersionKind:
+                  group: source.toolkit.fluxcd.io
+                  version: "v1beta2"
+                  kind: OCIRepository
+                metricNamePrefix: gotk
+                metrics:
+                  - name: "resource_info"
+                    help: "The current state of a GitOps Toolkit resource."
+                    each:
+                      type: Info
+                      info:
+                        labelsFromPath:
+                          name: [metadata, name]
+                    labelsFromPath:
+                      exported_namespace: [metadata, namespace]
+                      ready: [status, conditions, "[type=Ready]", status]
+              - groupVersionKind:
+                  group: kustomize.toolkit.fluxcd.io
+                  version: "v1"
+                  kind: Kustomization
+                metricNamePrefix: gotk
+                metrics:
+                  - name: "resource_info"
+                    help: "The current state of a GitOps Toolkit resource."
+                    each:
+                      type: Info
+                      info:
+                        labelsFromPath:
+                          name: [metadata, name]
+                    labelsFromPath:
+                      exported_namespace: [metadata, namespace]
+                      ready: [status, conditions, "[type=Ready]", status]
+              - groupVersionKind:
+                  group: helm.toolkit.fluxcd.io
+                  version: "v2beta1"
+                  kind: HelmRelease
+                metricNamePrefix: gotk
+                metrics:
+                  - name: "resource_info"
+                    help: "The current state of a GitOps Toolkit resource."
+                    each:
+                      type: Info
+                      info:
+                        labelsFromPath:
+                          name: [metadata, name]
+                    labelsFromPath:
+                      exported_namespace: [metadata, namespace]
+                      ready: [status, conditions, "[type=Ready]", status]
+              - groupVersionKind:
+                  group: image.toolkit.fluxcd.io
+                  version: "v1beta2"
+                  kind: ImageRepository
+                metricNamePrefix: gotk
+                metrics:
+                  - name: "resource_info"
+                    help: "The current state of a GitOps Toolkit resource."
+                    each:
+                      type: Info
+                      info:
+                        labelsFromPath:
+                          name: [metadata, name]
+                    labelsFromPath:
+                      exported_namespace: [metadata, namespace]
+                      ready: [status, conditions, "[type=Ready]", status]
+              - groupVersionKind:
+                  group: image.toolkit.fluxcd.io
+                  version: "v1beta2"
+                  kind: ImagePolicy
+                metricNamePrefix: gotk
+                metrics:
+                  - name: "resource_info"
+                    help: "The current state of a GitOps Toolkit resource."
+                    each:
+                      type: Info
+                      info:
+                        labelsFromPath:
+                          name: [metadata, name]
+                    labelsFromPath:
+                      exported_namespace: [metadata, namespace]
+                      ready: [status, conditions, "[type=Ready]", status]
+              - groupVersionKind:
+                  group: image.toolkit.fluxcd.io
+                  version: "v1beta1"
+                  kind: ImageUpdateAutomation
+                metricNamePrefix: gotk
+                metrics:
+                  - name: "resource_info"
+                    help: "The current state of a GitOps Toolkit resource."
+                    each:
+                      type: Info
+                      info:
+                        labelsFromPath:
+                          name: [metadata, name]
+                    labelsFromPath:
+                      exported_namespace: [metadata, namespace]
+                      ready: [status, conditions, "[type=Ready]", status]
+              - groupVersionKind:
+                  group: notification.toolkit.fluxcd.io
+                  version: "v1beta2"
+                  kind: Alert
+                metricNamePrefix: gotk
+                metrics:
+                  - name: "resource_info"
+                    help: "The current state of a GitOps Toolkit resource."
+                    each:
+                      type: Info
+                      info:
+                        labelsFromPath:
+                          name: [metadata, name]
+                    labelsFromPath:
+                      exported_namespace: [metadata, namespace]
+                      ready: [status, conditions, "[type=Ready]", status]
+              - groupVersionKind:
+                  group: notification.toolkit.fluxcd.io
+                  version: "v1beta2"
+                  kind: Provider
+                metricNamePrefix: gotk
+                metrics:
+                  - name: "resource_info"
+                    help: "The current state of a GitOps Toolkit resource."
+                    each:
+                      type: Info
+                      info:
+                        labelsFromPath:
+                          name: [metadata, name]
+                    labelsFromPath:
+                      exported_namespace: [metadata, namespace]
+                      ready: [status, conditions, "[type=Ready]", status]
+              - groupVersionKind:
+                  group: notification.toolkit.fluxcd.io
+                  version: "v1"
+                  kind: Receiver
+                metricNamePrefix: gotk
+                metrics:
+                  - name: "resource_info"
+                    help: "The current state of a GitOps Toolkit resource."
+                    each:
+                      type: Info
+                      info:
+                        labelsFromPath:
+                          name: [metadata, name]
+                    labelsFromPath:
+                      exported_namespace: [metadata, namespace]
+                      ready: [status, conditions, "[type=Ready]", status]
   postRenderers:
     - kustomize:
         patches:

manifests/monitoring/monitoring-config/dashboards/cluster.json

@@ -30,18 +30,23 @@
     ]
   },
   "editable": true,
-  "gnetId": null,
+  "fiscalYearStartMonth": 0,
   "graphTooltip": 0,
-  "iteration": 1652337714814,
+  "id": 5,
   "links": [],
+  "liveNow": false,
   "panels": [
     {
-      "datasource": "${DS_PROMETHEUS}",
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
       "description": "",
       "fieldConfig": {
         "defaults": {
           "decimals": 0,
           "mappings": [],
+          "noValue": "0",
           "thresholds": {
             "mode": "absolute",
             "steps": [
@@ -81,28 +86,37 @@
         "text": {},
         "textMode": "value"
       },
-      "pluginVersion": "7.5.5",
+      "pluginVersion": "10.0.2",
       "targets": [
         {
-          "exemplar": true,
+          "datasource": {
-          "expr": "count(gotk_reconcile_condition{namespace=~\"$operator_namespace\",exported_namespace=~\"$namespace\",type=\"Ready\",status=\"True\",kind=~\"Kustomization|HelmRelease\"})\n-\nsum(gotk_reconcile_condition{namespace=~\"$operator_namespace\",exported_namespace=~\"$namespace\",type=\"Ready\",status=\"Deleted\",kind=~\"Kustomization|HelmRelease\"})",
+            "type": "prometheus",
+            "uid": "prometheus"
+          },
+          "editorMode": "code",
+          "exemplar": false,
+          "expr": "count(gotk_resource_info{exported_namespace=~\"$namespace\", customresource_kind=~\"Kustomization|HelmRelease\"})",
+          "instant": true,
           "interval": "",
           "legendFormat": "",
+          "range": false,
           "refId": "A"
         }
       ],
-      "timeFrom": null,
-      "timeShift": null,
       "title": "Cluster Reconcilers",
       "type": "stat"
     },
     {
-      "datasource": "${DS_PROMETHEUS}",
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
       "description": "",
       "fieldConfig": {
         "defaults": {
           "decimals": 0,
           "mappings": [],
+          "noValue": "0",
           "thresholds": {
             "mode": "absolute",
             "steps": [
@@ -138,28 +152,37 @@
         "text": {},
         "textMode": "value"
       },
-      "pluginVersion": "7.5.5",
+      "pluginVersion": "10.0.2",
       "targets": [
         {
-          "exemplar": true,
+          "datasource": {
-          "expr": "sum(gotk_reconcile_condition{namespace=~\"$operator_namespace\",exported_namespace=~\"$namespace\",type=\"Ready\",status=\"False\",kind=~\"Kustomization|HelmRelease\"})",
+            "type": "prometheus",
+            "uid": "prometheus"
+          },
+          "editorMode": "code",
+          "exemplar": false,
+          "expr": "count(gotk_resource_info{exported_namespace=~\"$namespace\", customresource_kind=~\"Kustomization|HelmRelease\", ready=\"False\"})",
+          "instant": true,
           "interval": "",
           "legendFormat": "",
+          "range": false,
           "refId": "A"
         }
       ],
-      "timeFrom": null,
-      "timeShift": null,
       "title": "Failing Reconcilers",
       "type": "stat"
     },
     {
-      "datasource": "${DS_PROMETHEUS}",
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
       "description": "",
       "fieldConfig": {
         "defaults": {
           "decimals": 0,
           "mappings": [],
+          "noValue": "0",
           "thresholds": {
             "mode": "absolute",
             "steps": [
@@ -199,28 +222,37 @@
         "text": {},
         "textMode": "value"
       },
-      "pluginVersion": "7.5.5",
+      "pluginVersion": "10.0.2",
       "targets": [
         {
-          "exemplar": true,
+          "datasource": {
-          "expr": "count(gotk_reconcile_condition{namespace=~\"$operator_namespace\",exported_namespace=~\"$namespace\",type=\"Ready\",status=\"True\",kind=~\"GitRepository|HelmRepository|Bucket\"})\n-\nsum(gotk_reconcile_condition{namespace=~\"$operator_namespace\",exported_namespace=~\"$namespace\",type=\"Ready\",status=\"Deleted\",kind=~\"GitRepository|HelmRepository|Bucket\"})",
+            "type": "prometheus",
+            "uid": "prometheus"
+          },
+          "editorMode": "code",
+          "exemplar": false,
+          "expr": "count(gotk_resource_info{exported_namespace=~\"$namespace\", customresource_kind=~\"GitRepository|HelmRepository|Bucket|OCIRepository\"})",
+          "instant": true,
           "interval": "",
           "legendFormat": "",
+          "range": false,
           "refId": "A"
         }
       ],
-      "timeFrom": null,
-      "timeShift": null,
       "title": "Kubernetes Manifests Sources",
       "type": "stat"
     },
     {
-      "datasource": "${DS_PROMETHEUS}",
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
       "description": "",
       "fieldConfig": {
         "defaults": {
           "decimals": 0,
           "mappings": [],
+          "noValue": "0",
           "thresholds": {
             "mode": "absolute",
             "steps": [
@@ -256,18 +288,23 @@
         "text": {},
         "textMode": "value"
       },
-      "pluginVersion": "7.5.5",
+      "pluginVersion": "10.0.2",
       "targets": [
         {
-          "exemplar": true,
+          "datasource": {
-          "expr": "sum(gotk_reconcile_condition{namespace=~\"$operator_namespace\",exported_namespace=~\"$namespace\",type=\"Ready\",status=\"False\",kind=~\"GitRepository|HelmRepository|Bucket\"})",
+            "type": "prometheus",
+            "uid": "prometheus"
+          },
+          "editorMode": "code",
+          "exemplar": false,
+          "expr": "count(gotk_resource_info{exported_namespace=~\"$namespace\", customresource_kind=~\"GitRepository|HelmRepository|Bucket|OCIRepository\", ready=\"False\"})",
+          "instant": true,
           "interval": "",
           "legendFormat": "",
+          "range": false,
           "refId": "A"
         }
       ],
-      "timeFrom": null,
-      "timeShift": null,
       "title": "Failing Sources",
       "type": "stat"
     },
@@ -318,9 +355,10 @@
           "values": false
         },
         "showUnfilled": true,
-        "text": {}
+        "text": {},
+        "valueMode": "color"
       },
-      "pluginVersion": "7.5.5",
+      "pluginVersion": "10.0.2",
       "targets": [
         {
           "exemplar": true,
@@ -330,8 +368,6 @@
           "refId": "A"
         }
       ],
-      "timeFrom": null,
-      "timeShift": null,
       "title": "Reconciler ops avg. duration",
       "type": "bargauge"
     },
@@ -382,20 +418,19 @@
           "values": false
         },
         "showUnfilled": true,
-        "text": {}
+        "text": {},
+        "valueMode": "color"
       },
-      "pluginVersion": "7.5.5",
+      "pluginVersion": "10.0.2",
       "targets": [
         {
           "exemplar": true,
-          "expr": "  sum(rate(gotk_reconcile_duration_seconds_sum{namespace=~\"$operator_namespace\",exported_namespace=~\"$namespace\",kind=~\"GitRepository|HelmRepository|Bucket\"}[5m])) by (kind)\n/\n  sum(rate(gotk_reconcile_duration_seconds_count{namespace=~\"$operator_namespace\",exported_namespace=~\"$namespace\",kind=~\"GitRepository|HelmRepository|Bucket\"}[5m])) by (kind)",
+          "expr": "  sum(rate(gotk_reconcile_duration_seconds_sum{namespace=~\"$operator_namespace\",exported_namespace=~\"$namespace\",kind=~\"GitRepository|HelmRepository|Bucket|OCIRepository\"}[5m])) by (kind)\n/\n  sum(rate(gotk_reconcile_duration_seconds_count{namespace=~\"$operator_namespace\",exported_namespace=~\"$namespace\",kind=~\"GitRepository|HelmRepository|Bucket|OCIRepository\"}[5m])) by (kind)",
           "interval": "",
           "legendFormat": "{{kind}}",
           "refId": "A"
         }
       ],
-      "timeFrom": null,
-      "timeShift": null,
       "title": "Source ops avg. duration",
       "type": "bargauge"
     },
@@ -414,23 +449,33 @@
       "type": "row"
     },
     {
-      "datasource": "${DS_PROMETHEUS}",
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
       "description": "",
       "fieldConfig": {
         "defaults": {
           "custom": {
-            "displayMode": "auto",
+            "align": "auto",
+            "cellOptions": {
+              "type": "auto"
+            },
             "filterable": true,
             "inspect": false
           },
           "mappings": [
             {
               "options": {
-                "0": {
+                "False": {
-                  "text": "Ready"
+                  "color": "red",
-                },
+                  "index": 1,
-                "1": {
                   "text": "Not Ready"
+                },
+                "True": {
+                  "color": "blue",
+                  "index": 0,
+                  "text": "Ready"
                 }
               },
               "type": "value"
@@ -440,16 +485,8 @@
             "mode": "absolute",
             "steps": [
               {
-                "color": "blue",
+                "color": "transparent",
                 "value": null
-              },
-              {
-                "color": "blue",
-                "value": 0
-              },
-              {
-                "color": "red",
-                "value": 1
               }
             ]
           }
@@ -457,13 +494,16 @@
         "overrides": [
           {
             "matcher": {
-              "id": "byName",
+              "id": "byType",
-              "options": "Status"
+              "options": "string"
             },
             "properties": [
               {
-                "id": "custom.displayMode",
+                "id": "custom.cellOptions",
-                "value": "color-background"
+                "value": {
+                  "mode": "basic",
+                  "type": "color-background"
+                }
               }
             ]
           }
@@ -477,7 +517,9 @@
       },
       "id": 33,
       "options": {
+        "cellHeight": "sm",
         "footer": {
+          "countRows": false,
           "fields": "",
           "reducer": [
             "sum"
@@ -492,11 +534,16 @@
           }
         ]
       },
-      "pluginVersion": "7.5.5",
+      "pluginVersion": "10.0.2",
       "targets": [
         {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "prometheus"
+          },
+          "editorMode": "code",
           "exemplar": true,
-          "expr": "gotk_reconcile_condition{namespace=~\"$operator_namespace\",exported_namespace=~\"$namespace\",type=\"Ready\",status=\"False\",kind=~\"Kustomization|HelmRelease\"}",
+          "expr": "gotk_resource_info{exported_namespace=~\"$namespace\", customresource_kind=~\"Kustomization|HelmRelease\"}",
           "format": "table",
           "instant": true,
           "interval": "",
@@ -504,8 +551,6 @@
           "refId": "A"
         }
       ],
-      "timeFrom": null,
-      "timeShift": null,
       "title": "Cluster reconciliation readiness ",
       "transformations": [
         {
@@ -513,11 +558,16 @@
           "options": {
             "excludeByName": {
               "Time": true,
+              "Value": true,
               "__name__": true,
               "app": true,
               "container": true,
+              "customresource_group": true,
+              "customresource_kind": false,
+              "customresource_version": true,
               "endpoint": true,
               "exported_namespace": false,
+              "gotk_type": true,
               "instance": true,
               "job": true,
               "kubernetes_namespace": true,
@@ -525,16 +575,36 @@
               "namespace": true,
               "pod": true,
               "pod_template_hash": true,
+              "service": true,
               "status": true,
               "type": true
             },
-            "indexByName": {},
+            "indexByName": {
+              "Time": 0,
+              "Value": 15,
+              "__name__": 1,
+              "container": 2,
+              "customresource_group": 4,
+              "customresource_kind": 5,
+              "customresource_version": 6,
+              "endpoint": 7,
+              "exported_namespace": 3,
+              "instance": 8,
+              "job": 9,
+              "name": 10,
+              "namespace": 11,
+              "pod": 12,
+              "ready": 13,
+              "service": 14
+            },
             "renameByName": {
-              "Value": "Status",
+              "Value": "",
+              "customresource_kind": "Kind",
               "exported_namespace": "Namespace",
               "kind": "Kind",
               "name": "Name",
-              "namespace": "Operator Namespace"
+              "namespace": "Operator Namespace",
+              "ready": "Status"
             }
           }
         }
@@ -542,23 +612,36 @@
       "type": "table"
     },
     {
-      "datasource": "${DS_PROMETHEUS}",
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
       "description": "",
       "fieldConfig": {
         "defaults": {
+          "color": {
+            "mode": "thresholds"
+          },
           "custom": {
-            "displayMode": "auto",
+            "align": "auto",
+            "cellOptions": {
+              "type": "auto"
+            },
             "filterable": true,
             "inspect": false
           },
           "mappings": [
             {
               "options": {
-                "0": {
+                "False": {
-                  "text": "Ready"
+                  "color": "red",
-                },
+                  "index": 1,
-                "1": {
                   "text": "Not Ready"
+                },
+                "True": {
+                  "color": "blue",
+                  "index": 0,
+                  "text": "Ready"
                 }
               },
               "type": "value"
@@ -568,21 +651,28 @@
             "mode": "absolute",
             "steps": [
               {
-                "color": "blue",
+                "color": "transparent",
                 "value": null
-              },
-              {
-                "color": "blue",
-                "value": 0
-              },
-              {
-                "color": "red",
-                "value": 1
               }
             ]
           }
         },
         "overrides": [
+          {
+            "matcher": {
+              "id": "byType",
+              "options": "string"
+            },
+            "properties": [
+              {
+                "id": "custom.cellOptions",
+                "value": {
+                  "mode": "basic",
+                  "type": "color-background"
+                }
+              }
+            ]
+          },
           {
             "matcher": {
               "id": "byName",
@@ -590,8 +680,15 @@
             },
             "properties": [
               {
-                "id": "custom.displayMode",
+                "id": "noValue",
-                "value": "color-background"
+                "value": "Ready"
+              },
+              {
+                "id": "color",
+                "value": {
+                  "fixedColor": "blue",
+                  "mode": "fixed"
+                }
               }
             ]
           }
@@ -605,7 +702,9 @@
       },
       "id": 34,
       "options": {
+        "cellHeight": "sm",
         "footer": {
+          "countRows": false,
           "fields": "",
           "reducer": [
             "sum"
@@ -620,11 +719,16 @@
           }
         ]
       },
-      "pluginVersion": "7.5.5",
+      "pluginVersion": "10.0.2",
       "targets": [
         {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "prometheus"
+          },
+          "editorMode": "code",
           "exemplar": true,
-          "expr": "gotk_reconcile_condition{namespace=~\"$operator_namespace\",exported_namespace=~\"$namespace\",type=\"Ready\",status=\"False\",kind=~\"GitRepository|HelmRepository|Bucket\"}",
+          "expr": "gotk_resource_info{exported_namespace=~\"$namespace\", customresource_kind=~\"GitRepository|HelmRepository|Bucket|OCIRepository\"}",
           "format": "table",
           "instant": true,
           "interval": "",
@@ -632,8 +736,6 @@
           "refId": "A"
         }
       ],
-      "timeFrom": null,
-      "timeShift": null,
       "title": "Source acquisition readiness ",
       "transformations": [
         {
@@ -641,11 +743,16 @@
           "options": {
             "excludeByName": {
               "Time": true,
+              "Value": true,
               "__name__": true,
               "app": true,
               "container": true,
+              "customresource_group": true,
+              "customresource_kind": false,
+              "customresource_version": true,
               "endpoint": true,
               "exported_namespace": false,
+              "gotk_type": true,
               "instance": true,
               "job": true,
               "kubernetes_namespace": true,
@@ -653,16 +760,37 @@
               "namespace": true,
               "pod": true,
               "pod_template_hash": true,
+              "ready": false,
+              "service": true,
               "status": true,
               "type": true
             },
-            "indexByName": {},
+            "indexByName": {
+              "Time": 0,
+              "Value": 15,
+              "__name__": 1,
+              "container": 2,
+              "customresource_group": 5,
+              "customresource_kind": 6,
+              "customresource_version": 7,
+              "endpoint": 8,
+              "exported_namespace": 4,
+              "instance": 9,
+              "job": 10,
+              "name": 11,
+              "namespace": 3,
+              "pod": 12,
+              "ready": 13,
+              "service": 14
+            },
             "renameByName": {
-              "Value": "Status",
+              "Value": "",
+              "customresource_kind": "Kind",
               "exported_namespace": "Namespace",
               "kind": "Kind",
               "name": "Name",
-              "namespace": "Operator Namespace"
+              "namespace": "Operator Namespace",
+              "ready": "Status"
             }
           }
         }
@@ -690,10 +818,6 @@
       "dashes": false,
       "datasource": "${DS_PROMETHEUS}",
       "description": "",
-      "fieldConfig": {
-        "defaults": {},
-        "overrides": []
-      },
       "fill": 1,
       "fillGradient": 0,
       "gridPos": {
@@ -724,7 +848,7 @@
         "alertThreshold": true
       },
       "percentage": false,
-      "pluginVersion": "7.5.5",
+      "pluginVersion": "10.0.2",
       "pointradius": 2,
       "points": false,
       "renderer": "flot",
@@ -743,9 +867,7 @@
         }
       ],
       "thresholds": [],
-      "timeFrom": null,
       "timeRegions": [],
-      "timeShift": null,
       "title": "Cluster reconciliation duration",
       "tooltip": {
         "shared": true,
@@ -754,33 +876,24 @@
       },
       "type": "graph",
       "xaxis": {
-        "buckets": null,
         "mode": "time",
-        "name": null,
         "show": true,
         "values": []
       },
       "yaxes": [
         {
           "format": "s",
-          "label": null,
           "logBase": 1,
-          "max": null,
-          "min": null,
           "show": true
         },
         {
           "format": "short",
-          "label": null,
           "logBase": 1,
-          "max": null,
-          "min": null,
           "show": true
         }
       ],
       "yaxis": {
-        "align": false,
+        "align": false
-        "alignLevel": null
       }
     },
     {
@@ -790,10 +903,6 @@
       "dashes": false,
       "datasource": "${DS_PROMETHEUS}",
       "description": "",
-      "fieldConfig": {
-        "defaults": {},
-        "overrides": []
-      },
       "fill": 1,
       "fillGradient": 0,
       "gridPos": {
@@ -824,7 +933,7 @@
         "alertThreshold": true
       },
       "percentage": false,
-      "pluginVersion": "7.5.5",
+      "pluginVersion": "10.0.2",
       "pointradius": 2,
       "points": false,
       "renderer": "flot",
@@ -835,7 +944,7 @@
       "targets": [
         {
           "exemplar": true,
-          "expr": "  sum(rate(gotk_reconcile_duration_seconds_sum{namespace=~\"$operator_namespace\",exported_namespace=~\"$namespace\",kind=~\"GitRepository|HelmRepository|Bucket\"}[5m])) by (kind, name)\n/\n  sum(rate(gotk_reconcile_duration_seconds_count{namespace=~\"$operator_namespace\",exported_namespace=~\"$namespace\",kind=~\"GitRepository|HelmRepository|Bucket\"}[5m])) by (kind, name)",
+          "expr": "  sum(rate(gotk_reconcile_duration_seconds_sum{namespace=~\"$operator_namespace\",exported_namespace=~\"$namespace\",kind=~\"GitRepository|HelmRepository|Bucket|OCIRepository\"}[5m])) by (kind, name)\n/\n  sum(rate(gotk_reconcile_duration_seconds_count{namespace=~\"$operator_namespace\",exported_namespace=~\"$namespace\",kind=~\"GitRepository|HelmRepository|Bucket|OCIRepository\"}[5m])) by (kind, name)",
           "hide": false,
           "interval": "",
           "legendFormat": "{{kind}}/{{name}}",
@@ -843,9 +952,7 @@
         }
       ],
       "thresholds": [],
-      "timeFrom": null,
       "timeRegions": [],
-      "timeShift": null,
       "title": "Source acquisition duration",
       "tooltip": {
         "shared": true,
@@ -854,38 +961,29 @@
       },
       "type": "graph",
       "xaxis": {
-        "buckets": null,
         "mode": "time",
-        "name": null,
         "show": true,
         "values": []
       },
       "yaxes": [
         {
           "format": "s",
-          "label": null,
           "logBase": 1,
-          "max": null,
-          "min": null,
           "show": true
         },
         {
           "format": "short",
-          "label": null,
           "logBase": 1,
-          "max": null,
-          "min": null,
           "show": true
         }
       ],
       "yaxis": {
-        "align": false,
+        "align": false
-        "alignLevel": null
       }
     }
   ],
   "refresh": "30s",
-  "schemaVersion": 36,
+  "schemaVersion": 38,
   "style": "light",
   "tags": [
     "flux"
@@ -903,13 +1001,13 @@
             "$__all"
           ]
         },
-        "datasource": "$DS_PROMETHEUS",
+        "datasource": {
+          "type": "prometheus",
+          "uid": "$DS_PROMETHEUS"
+        },
         "definition": "label_values(gotk_reconcile_condition, namespace)",
-        "description": null,
-        "error": null,
         "hide": 0,
         "includeAll": true,
-        "label": null,
         "multi": true,
         "name": "operator_namespace",
         "options": [],
@@ -928,10 +1026,8 @@
         "useTags": false
       },
       {
-        "allValue": null,
         "current": {
           "selected": true,
-          "tags": [],
           "text": [
             "All"
           ],
@@ -939,19 +1035,19 @@
             "$__all"
           ]
         },
-        "datasource": "$DS_PROMETHEUS",
+        "datasource": {
-        "definition": "label_values(gotk_reconcile_condition, exported_namespace)",
+          "type": "prometheus",
-        "description": null,
+          "uid": "$DS_PROMETHEUS"
-        "error": null,
+        },
+        "definition": "label_values(gotk_resource_info,exported_namespace)",
         "hide": 0,
         "includeAll": true,
-        "label": null,
         "multi": true,
         "name": "namespace",
         "options": [],
         "query": {
-          "query": "label_values(gotk_reconcile_condition, exported_namespace)",
+          "query": "label_values(gotk_resource_info,exported_namespace)",
-          "refId": "StandardVariableQuery"
+          "refId": "PrometheusVariableQueryEditor-VariableQuery"
         },
         "refresh": 2,
         "regex": "",
@@ -1000,7 +1096,9 @@
       "1d"
     ]
   },
+  "timezone": "",
   "title": "Flux Cluster Stats",
   "uid": "flux-cluster",
-  "version": 3
+  "version": 4,
+  "weekStart": ""
 }