From d76c34fa83bed586de26ba05971031d4141d3393 Mon Sep 17 00:00:00 2001 From: Alison Dowdney Date: Thu, 15 Apr 2021 22:31:46 +0100 Subject: [PATCH] Break out Notification Guide Signed-off-by: Alison Dowdney --- docs/guides/chat-notifications.md | 187 +++++++++++++++++++++ docs/guides/commit-notifications.md | 241 ++++++++++++++++++++++++++++ mkdocs.yml | 4 +- 3 files changed, 431 insertions(+), 1 deletion(-) create mode 100644 docs/guides/chat-notifications.md create mode 100644 docs/guides/commit-notifications.md diff --git a/docs/guides/chat-notifications.md b/docs/guides/chat-notifications.md new file mode 100644 index 00000000..7a1649e8 --- /dev/null +++ b/docs/guides/chat-notifications.md @@ -0,0 +1,187 @@ +# Chat based notifications + +When operating a cluster, you may wish to receive notifications about the clusters status in a messaging service. + +For example, the status of a deployment, GitResource, or Kustomization. + +This guide will walk you through how to setup Chat based notifications, for a variety of messaging providers. + +## Prerequisites + +To follow this guide you'll need a Kubernetes cluster with the GitOps +toolkit controllers installed on it. +Please see the [get started guide](../get-started/index.md) +or the [installation guide](installation.md). + +The GitOps toolkit controllers emit Kubernetes events whenever a resource status changes. +You can use the [notification-controller](../components/notification/controller.md) +to forward these events to Slack, Microsoft Teams, Discord or Rocket chart. +The notification controller is part of the default toolkit installation. + + +## Define a provider + +First create a secret with your incoming webhook: + +```sh +kubectl -n flux-system create secret generic webhook-url \ +--from-literal=address=https://webhooks.com/services/YOUR/WEBHOOK +``` + +Note that the secret must contain an `address` field, + +it can be a Slack, Microsoft Teams, Discord, Google Chat or Rocket webhook URL. + +Create a notification provider for your service by referencing the above secret: + +=== "Slack" + ```bash + $ flux create alert-provider slack \ + --type slack \ + --secret-ref webhook-url \ + --channel general \ + --export > slack-provider.yaml + --- + # slack-provider.yaml + apiVersion: notification.toolkit.fluxcd.io/v1beta1 + kind: Provider + metadata: + name: slack + namespace: flux-system + spec: + type: slack + channel: general + secretRef: + name: webhook-url + ``` + +=== "Discord" + + ```bash + $ flux create alert-provider discord \ + --type discord \ + --secret-ref webhook-url \ + --channel general \ + --username flux \ + --export > discord-provider.yaml + --- + # discord-provider.yaml + apiVersion: notification.toolkit.fluxcd.io/v1beta1 + kind: Provider + metadata: + name: discord + namespace: flux-system + spec: + channel: notifications-alerts-webhooks-example + secretRef: + name: webhook-url + type: discord + username: notifications-alerts-webhooks-example-bot + + ``` +=== "Microsoft Teams" + ```bash + $ flux create alert-provider teams \ + --type msteams \ + --secret-ref webhook-url \ + --channel general \ + --export > teams-provider.yaml + --- + # teams-provider.yaml + apiVersion: notification.toolkit.fluxcd.io/v1beta1 + kind: Provider + metadata: + name: teams + namespace: flux-system + spec: + channel: general + secretRef: + name: webhook-url + type: msteams + --- +=== "Google Chat" + ```bash + $ flux create alert-provider google-chat \ + --type googlechat \ + --secret-ref webnook-url \ + --export > gchat-provider.yaml + --- + # gchat-provider.yaml + apiVersion: notification.toolkit.fluxcd.io/v1beta1 + kind: Provider + metadata: + name: google-chat + namespace: flux-system + spec: + secretRef: + name: webhook-url + type: googlechat + ``` + + +The provider type can be `slack`, `discord`, `msteams`, `rocket`, `googlechat` or `generic` + +When type `generic` is specified, the notification controller will post the incoming +[event](../components/notification/event.md) in JSON format to the webhook address. +This way you can create custom handlers that can store the events in +Elasticsearch, CloudWatch, Stackdriver, etc. + +## Define an alert + +Create an alert definition for all repositories and kustomizations: + +```bash +flux create alert teams-alert \ +--provider-ref teams \ +--event-severity info \ +--event-source Kustomization/'*' \ +--event-source GitRepository/'*' \ +--namespace flux-system + +``` +```yaml +apiVersion: notification.toolkit.fluxcd.io/v1beta1 +kind: Alert +metadata: + name: on-call-webapp + namespace: flux-system +spec: + providerRef: + name: slack + eventSeverity: info + eventSources: + - kind: GitRepository + name: '*' + - kind: Kustomization + name: '*' +``` + +Apply the above files or commit them to the `fleet-infra` repository. + +To verify that the alert has been acknowledge by the notification controller do: + +```console +$ kubectl -n flux-system get alerts + +NAME READY STATUS AGE +on-call-webapp True Initialized 1m +``` + +Multiple alerts can be used to send notifications to different channels or Slack organizations. + +The event severity can be set to `info` or `error`. +When the severity is set to `error`, the kustomize controller will alert on any error +encountered during the reconciliation process. +This includes kustomize build and validation errors, +apply errors and health check failures. + +![error alert](../_files/slack-error-alert.png) + +When the verbosity is set to `info`, the controller will alert if: + +* a Kubernetes object was created, updated or deleted +* heath checks are passing +* a dependency is delaying the execution +* an error occurs + +![info alert](../_files/slack-info-alert.png) diff --git a/docs/guides/commit-notifications.md b/docs/guides/commit-notifications.md new file mode 100644 index 00000000..0ea2fd56 --- /dev/null +++ b/docs/guides/commit-notifications.md @@ -0,0 +1,241 @@ +# Commit based Notifications + +The GitHub, GitLab, Bitbucket, and Azure DevOps providers are slightly different to the other providers. Instead of +a stateless stream of events, the git notification providers will link the event with accompanying git commit which +triggered the event. The linking is done by updating the commit status of a specific commit. + + - [GitHub](https://docs.github.com/en/github/collaborating-with-issues-and-pull-requests/about-status-checks) + - [GitLab](https://docs.gitlab.com/ee/api/commits.html) + - [Bitbucket](https://developer.atlassian.com/server/bitbucket/how-tos/updating-build-status-for-commits/) + - [Azure DevOps](https://docs.microsoft.com/en-us/rest/api/azure/devops/git/statuses?view=azure-devops-rest-6.0) + +In GitHub the commit status set by notification-controller will result in a green checkmark or red cross next to the commit hash. +Clicking the icon will show more detailed information about the status. +![commit status GitHub overview](../_files/commit-status-github-overview.png) + +Receiving an event in the form of a commit status has the benefit that it closes the deployment loop. This gives quick and visible feedback if a commit has reconciled and if it succeeded. This means deployments operate in a similar manner to "traditional" push based CD. +Status can be fetched from the git providers API for a specific commit. This allows for custom automation tools that can automatically promote, commit to a new directory, after receiving a successful commit status. This can all be done without requiring any access to the Kubernetes cluster + +The provider works by referencing the same git repository as the Kustomization controller does. +When a new commit is pushed to the repository, source-controller will sync the commit, triggering the kustomize-controller +to reconcile the new commit. After this is done the kustomize-controller sends an event to the notification-controller +with the result and the commit hash it reconciled. Then notification-controller can update the correct commit and repository when receiving the event. + + +![commit status flow](../_files/commit-status-flow.png) + +!!! hint "Limitations" + The git notification providers require that a commit hash present in the meta data + of the event. There for the the providers will only work with `Kustomization` as an + event source, as it is the only resource which includes this data. + +## Prerequisites + +### Authentication Token + +#### Obtaining your git providers credentials + +The token/app-password will need to have write access for the repository it will be updating the commit status in. + +Follow the respective guide for your git provider + +- [GitHub personal access token](https://docs.github.com/en/free-pro-team@latest/github/authenticating-to-github/creating-a-personal-access-token) +- [GitLab personal access token](https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html) +- [Azure DevOps personal access token](https://docs.microsoft.com/en-us/azure/devops/organizations/accounts/use-personal-access-tokens-to-authenticate?view=azure-devops&tabs=preview-page) +- [app password](https://support.atlassian.com/bitbucket-cloud/docs/app-passwords/) + +#### Creating Git Secret for your token/app-password + +Use the following command to store the token/app-password in a secret + +=== "GitHub, GitLab, Azure DevOps" + ```bash + $ kubectl -n flux-system create secret generic git \ + --from-literal=token='' + --- + apiVersion: v1 + data: + token: PFlPVVJUT0tFTj4= + kind: Secret + metadata: + creationTimestamp: null + name: git + namespace: flux-system + ``` +=== "BitBucket" + ```bash + $ kubectl -n flux-system create secret generic bitbucket\ + --from-literal=token=':' + --- + apiVersion: v1 + kind: Secret + metadata: + name: git + namespace: default + data: + token: : + ``` + +## Creating a Deployment + +Copy the manifest content in the "[kustomize](https://github.com/stefanprodan/podinfo/tree/master/kustomize)" directory +into the directory "./clusters/my-cluster/podinfo" in your fleet-infra repository. Make sure that you also add the +namespace podinfo. + +```yaml +apiVersion: v1 +kind: Namespace +metadata: + name: podinfo +``` + +Then create a Kustomization to deploy podinfo. +```yaml +apiVersion: kustomize.toolkit.fluxcd.io/v1beta1 +kind: Kustomization +metadata: + name: podinfo + namespace: flux-system +spec: + interval: 5m + targetNamespace: podinfo + path: ./clusters/my-cluster/podinfo + prune: true + sourceRef: + kind: GitRepository + name: flux-system + healthChecks: + - apiVersion: apps/v1 + kind: Deployment + name: podinfo + namespace: podinfo + timeout: 1m +``` + +## Creating the Git provider + +Creating a git provider is similar to creating other providers. +The only difference is that with git providers `spec.address` needs to point to the same git repository that the event source originates from. + +=== "GitHub" + ```yaml + apiVersion: notification.toolkit.fluxcd.io/v1beta1 + kind: Provider + metadata: + name: flux-system + namespace: flux-system + spec: + type: github + address: https://github.com//fleet-infra + secretRef: + name: git + ``` + +=== "GitLab" + ```yaml + apiVersion: notification.toolkit.fluxcd.io/v1beta1 + kind: Provider + metadata: + name: flux-system + namespace: flux-system + spec: + type: gitlab + address: https://gitlab.com//fleet-infra + secretRef: + name: git + ``` +=== "Bitbucket" + ```bash + ``` +=== "Azure DevOps" + ```bash + ``` + +## Creating the alert + +=== "GitHub" + ```bash + $ flux create alert podinfo \ + --provider-ref git \ + --event-severity info \ + --event-source 'Kustomization/podinfo' \ + --namespace 'flux-system' \ + --export + --- + apiVersion: notification.toolkit.fluxcd.io/v1beta1 + kind: Alert + metadata: + name: podinfo + namespace: flux-system + spec: + providerRef: + name: flux-system + eventSeverity: info + eventSources: + - kind: Kustomization + name: podinfo + namespace: flux-system + ``` +=== "GitLab" + ```bash + ``` +=== "Bitbucket" + ```bash + ``` +=== "Azure DevOps" + ```bash + ``` + + +By now the fleet-infra repository should have a similar directory structure. +``` +fleet-infra +└── clusters/ + └── my-cluster/ + ├── flux-system/ + │ ├── gotk-components.yaml + │ ├── gotk-sync.yaml + │ └── kustomization.yaml + ├── podinfo/ + │ ├── namespace.yaml + │ ├── deployment.yaml + │ ├── hpa.yaml + │ ├── service.yaml + │ └── kustomization.yaml + ├── podinfo-kustomization.yaml + └── podinfo-notification.yaml +``` + +If podinfo is deployed and the health checks pass you should get a successful status in +your forked podinfo repository. + +If everything is setup correctly there should now be a green check-mark next to the latest commit. +Clicking the check-mark should show a detailed view. + +=== "Github" + ![commit status GitHub successful](../_files/commit-status-github-success.png) +=== "GitLab" + ![commit status GitLab successful](../_files/commit-status-gitlab-success.png) | + +## Generate an error + +A deployment failure can be forced by setting an invalid image tag in the podinfo deployment. +```yaml +apiVersion: apps/v1 +kind: Deployment +spec: + template: + spec: + containers: + - name: podinfod + image: ghcr.io/stefanprodan/podinfo:fake +``` + +After the commit has been reconciled it should return a failed commit status. + +=== "GitHub" + ![commit status GitHub failure](../_files/commit-status-github-failure.png) +=== "GitLab" + ![commit status GitLab failure](../_files/commit-status-gitlab-failure.png) | + + diff --git a/mkdocs.yml b/mkdocs.yml index feab9507..18d57b8d 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -57,7 +57,9 @@ nav: - Guides: - Installation: guides/installation.md - Manage Helm Releases: guides/helmreleases.md - - Setup Notifications: guides/notifications.md + - Setup Chat Based Notifications: guides/chat-notifications.md + - Setup Commit Based Notifications: guides/commit-notifications.md + - Notifications: guides/notifications.md - Setup Webhook Receivers: guides/webhook-receivers.md - Monitoring with Prometheus: guides/monitoring.md - Sealed Secrets: guides/sealed-secrets.md