mirror of https://github.com/fluxcd/flux2.git
Break out Notification Guide
Signed-off-by: Alison Dowdney <alison@alisondowdney.com>pull/1279/head
Alison Dowdney
4 years ago
parent
7dcf884e38
commit
d76c34fa83
@ -0,0 +1,187 @@
|
|||||||
|
# Chat based notifications
|
||||||
|
|
||||||
|
When operating a cluster, you may wish to receive notifications about the clusters status in a messaging service.
|
||||||
|
|
||||||
|
For example, the status of a deployment, GitResource, or Kustomization.
|
||||||
|
|
||||||
|
This guide will walk you through how to setup Chat based notifications, for a variety of messaging providers.
|
||||||
|
|
||||||
|
## Prerequisites
|
||||||
|
|
||||||
|
To follow this guide you'll need a Kubernetes cluster with the GitOps
|
||||||
|
toolkit controllers installed on it.
|
||||||
|
Please see the [get started guide](../get-started/index.md)
|
||||||
|
or the [installation guide](installation.md).
|
||||||
|
|
||||||
|
The GitOps toolkit controllers emit Kubernetes events whenever a resource status changes.
|
||||||
|
You can use the [notification-controller](../components/notification/controller.md)
|
||||||
|
to forward these events to Slack, Microsoft Teams, Discord or Rocket chart.
|
||||||
|
The notification controller is part of the default toolkit installation.
|
||||||
|
|
||||||
|
|
||||||
|
## Define a provider
|
||||||
|
|
||||||
|
First create a secret with your incoming webhook:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
kubectl -n flux-system create secret generic webhook-url \
|
||||||
|
--from-literal=address=https://webhooks.com/services/YOUR/WEBHOOK
|
||||||
|
```
|
||||||
|
|
||||||
|
Note that the secret must contain an `address` field,
|
||||||
|
|
||||||
|
it can be a Slack, Microsoft Teams, Discord, Google Chat or Rocket webhook URL.
|
||||||
|
|
||||||
|
Create a notification provider for your service by referencing the above secret:
|
||||||
|
|
||||||
|
=== "Slack"
|
||||||
|
```bash
|
||||||
|
$ flux create alert-provider slack \
|
||||||
|
--type slack \
|
||||||
|
--secret-ref webhook-url \
|
||||||
|
--channel general \
|
||||||
|
--export > slack-provider.yaml
|
||||||
|
---
|
||||||
|
# slack-provider.yaml
|
||||||
|
apiVersion: notification.toolkit.fluxcd.io/v1beta1
|
||||||
|
kind: Provider
|
||||||
|
metadata:
|
||||||
|
name: slack
|
||||||
|
namespace: flux-system
|
||||||
|
spec:
|
||||||
|
type: slack
|
||||||
|
channel: general
|
||||||
|
secretRef:
|
||||||
|
name: webhook-url
|
||||||
|
```
|
||||||
|
|
||||||
|
=== "Discord"
|
||||||
|
|
||||||
|
```bash
|
||||||
|
$ flux create alert-provider discord \
|
||||||
|
--type discord \
|
||||||
|
--secret-ref webhook-url \
|
||||||
|
--channel general \
|
||||||
|
--username flux \
|
||||||
|
--export > discord-provider.yaml
|
||||||
|
---
|
||||||
|
# discord-provider.yaml
|
||||||
|
apiVersion: notification.toolkit.fluxcd.io/v1beta1
|
||||||
|
kind: Provider
|
||||||
|
metadata:
|
||||||
|
name: discord
|
||||||
|
namespace: flux-system
|
||||||
|
spec:
|
||||||
|
channel: notifications-alerts-webhooks-example
|
||||||
|
secretRef:
|
||||||
|
name: webhook-url
|
||||||
|
type: discord
|
||||||
|
username: notifications-alerts-webhooks-example-bot
|
||||||
|
|
||||||
|
```
|
||||||
|
=== "Microsoft Teams"
|
||||||
|
```bash
|
||||||
|
$ flux create alert-provider teams \
|
||||||
|
--type msteams \
|
||||||
|
--secret-ref webhook-url \
|
||||||
|
--channel general \
|
||||||
|
--export > teams-provider.yaml
|
||||||
|
---
|
||||||
|
# teams-provider.yaml
|
||||||
|
apiVersion: notification.toolkit.fluxcd.io/v1beta1
|
||||||
|
kind: Provider
|
||||||
|
metadata:
|
||||||
|
name: teams
|
||||||
|
namespace: flux-system
|
||||||
|
spec:
|
||||||
|
channel: general
|
||||||
|
secretRef:
|
||||||
|
name: webhook-url
|
||||||
|
type: msteams
|
||||||
|
---
|
||||||
|
=== "Google Chat"
|
||||||
|
```bash
|
||||||
|
$ flux create alert-provider google-chat \
|
||||||
|
--type googlechat \
|
||||||
|
--secret-ref webnook-url \
|
||||||
|
--export > gchat-provider.yaml
|
||||||
|
---
|
||||||
|
# gchat-provider.yaml
|
||||||
|
apiVersion: notification.toolkit.fluxcd.io/v1beta1
|
||||||
|
kind: Provider
|
||||||
|
metadata:
|
||||||
|
name: google-chat
|
||||||
|
namespace: flux-system
|
||||||
|
spec:
|
||||||
|
secretRef:
|
||||||
|
name: webhook-url
|
||||||
|
type: googlechat
|
||||||
|
```
|
||||||
|
|
||||||
|
|
||||||
|
The provider type can be `slack`, `discord`, `msteams`, `rocket`, `googlechat` or `generic`
|
||||||
|
|
||||||
|
When type `generic` is specified, the notification controller will post the incoming
|
||||||
|
[event](../components/notification/event.md) in JSON format to the webhook address.
|
||||||
|
This way you can create custom handlers that can store the events in
|
||||||
|
Elasticsearch, CloudWatch, Stackdriver, etc.
|
||||||
|
|
||||||
|
## Define an alert
|
||||||
|
|
||||||
|
Create an alert definition for all repositories and kustomizations:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
flux create alert teams-alert \
|
||||||
|
--provider-ref teams \
|
||||||
|
--event-severity info \
|
||||||
|
--event-source Kustomization/'*' \
|
||||||
|
--event-source GitRepository/'*' \
|
||||||
|
--namespace flux-system
|
||||||
|
|
||||||
|
```
|
||||||
|
```yaml
|
||||||
|
apiVersion: notification.toolkit.fluxcd.io/v1beta1
|
||||||
|
kind: Alert
|
||||||
|
metadata:
|
||||||
|
name: on-call-webapp
|
||||||
|
namespace: flux-system
|
||||||
|
spec:
|
||||||
|
providerRef:
|
||||||
|
name: slack
|
||||||
|
eventSeverity: info
|
||||||
|
eventSources:
|
||||||
|
- kind: GitRepository
|
||||||
|
name: '*'
|
||||||
|
- kind: Kustomization
|
||||||
|
name: '*'
|
||||||
|
```
|
||||||
|
|
||||||
|
Apply the above files or commit them to the `fleet-infra` repository.
|
||||||
|
|
||||||
|
To verify that the alert has been acknowledge by the notification controller do:
|
||||||
|
|
||||||
|
```console
|
||||||
|
$ kubectl -n flux-system get alerts
|
||||||
|
|
||||||
|
NAME READY STATUS AGE
|
||||||
|
on-call-webapp True Initialized 1m
|
||||||
|
```
|
||||||
|
|
||||||
|
Multiple alerts can be used to send notifications to different channels or Slack organizations.
|
||||||
|
|
||||||
|
The event severity can be set to `info` or `error`.
|
||||||
|
When the severity is set to `error`, the kustomize controller will alert on any error
|
||||||
|
encountered during the reconciliation process.
|
||||||
|
This includes kustomize build and validation errors,
|
||||||
|
apply errors and health check failures.
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
When the verbosity is set to `info`, the controller will alert if:
|
||||||
|
|
||||||
|
* a Kubernetes object was created, updated or deleted
|
||||||
|
* heath checks are passing
|
||||||
|
* a dependency is delaying the execution
|
||||||
|
* an error occurs
|
||||||
|
|
||||||
|
|
||||||
@ -0,0 +1,241 @@
|
|||||||
|
# Commit based Notifications
|
||||||
|
|
||||||
|
The GitHub, GitLab, Bitbucket, and Azure DevOps providers are slightly different to the other providers. Instead of
|
||||||
|
a stateless stream of events, the git notification providers will link the event with accompanying git commit which
|
||||||
|
triggered the event. The linking is done by updating the commit status of a specific commit.
|
||||||
|
|
||||||
|
- [GitHub](https://docs.github.com/en/github/collaborating-with-issues-and-pull-requests/about-status-checks)
|
||||||
|
- [GitLab](https://docs.gitlab.com/ee/api/commits.html)
|
||||||
|
- [Bitbucket](https://developer.atlassian.com/server/bitbucket/how-tos/updating-build-status-for-commits/)
|
||||||
|
- [Azure DevOps](https://docs.microsoft.com/en-us/rest/api/azure/devops/git/statuses?view=azure-devops-rest-6.0)
|
||||||
|
|
||||||
|
In GitHub the commit status set by notification-controller will result in a green checkmark or red cross next to the commit hash.
|
||||||
|
Clicking the icon will show more detailed information about the status.
|
||||||
|
|
||||||
|
|
||||||
|
Receiving an event in the form of a commit status has the benefit that it closes the deployment loop. This gives quick and visible feedback if a commit has reconciled and if it succeeded. This means deployments operate in a similar manner to "traditional" push based CD.
|
||||||
|
Status can be fetched from the git providers API for a specific commit. This allows for custom automation tools that can automatically promote, commit to a new directory, after receiving a successful commit status. This can all be done without requiring any access to the Kubernetes cluster
|
||||||
|
|
||||||
|
The provider works by referencing the same git repository as the Kustomization controller does.
|
||||||
|
When a new commit is pushed to the repository, source-controller will sync the commit, triggering the kustomize-controller
|
||||||
|
to reconcile the new commit. After this is done the kustomize-controller sends an event to the notification-controller
|
||||||
|
with the result and the commit hash it reconciled. Then notification-controller can update the correct commit and repository when receiving the event.
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
!!! hint "Limitations"
|
||||||
|
The git notification providers require that a commit hash present in the meta data
|
||||||
|
of the event. There for the the providers will only work with `Kustomization` as an
|
||||||
|
event source, as it is the only resource which includes this data.
|
||||||
|
|
||||||
|
## Prerequisites
|
||||||
|
|
||||||
|
### Authentication Token
|
||||||
|
|
||||||
|
#### Obtaining your git providers credentials
|
||||||
|
|
||||||
|
The token/app-password will need to have write access for the repository it will be updating the commit status in.
|
||||||
|
|
||||||
|
Follow the respective guide for your git provider
|
||||||
|
|
||||||
|
- [GitHub personal access token](https://docs.github.com/en/free-pro-team@latest/github/authenticating-to-github/creating-a-personal-access-token)
|
||||||
|
- [GitLab personal access token](https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html)
|
||||||
|
- [Azure DevOps personal access token](https://docs.microsoft.com/en-us/azure/devops/organizations/accounts/use-personal-access-tokens-to-authenticate?view=azure-devops&tabs=preview-page)
|
||||||
|
- [app password](https://support.atlassian.com/bitbucket-cloud/docs/app-passwords/)
|
||||||
|
|
||||||
|
#### Creating Git Secret for your token/app-password
|
||||||
|
|
||||||
|
Use the following command to store the token/app-password in a secret
|
||||||
|
|
||||||
|
=== "GitHub, GitLab, Azure DevOps"
|
||||||
|
```bash
|
||||||
|
$ kubectl -n flux-system create secret generic git \
|
||||||
|
--from-literal=token='<YOURTOKEN>'
|
||||||
|
---
|
||||||
|
apiVersion: v1
|
||||||
|
data:
|
||||||
|
token: PFlPVVJUT0tFTj4=
|
||||||
|
kind: Secret
|
||||||
|
metadata:
|
||||||
|
creationTimestamp: null
|
||||||
|
name: git
|
||||||
|
namespace: flux-system
|
||||||
|
```
|
||||||
|
=== "BitBucket"
|
||||||
|
```bash
|
||||||
|
$ kubectl -n flux-system create secret generic bitbucket\
|
||||||
|
--from-literal=token='<username>:<app-password>'
|
||||||
|
---
|
||||||
|
apiVersion: v1
|
||||||
|
kind: Secret
|
||||||
|
metadata:
|
||||||
|
name: git
|
||||||
|
namespace: default
|
||||||
|
data:
|
||||||
|
token: <username>:<app-password>
|
||||||
|
```
|
||||||
|
|
||||||
|
## Creating a Deployment
|
||||||
|
|
||||||
|
Copy the manifest content in the "[kustomize](https://github.com/stefanprodan/podinfo/tree/master/kustomize)" directory
|
||||||
|
into the directory "./clusters/my-cluster/podinfo" in your fleet-infra repository. Make sure that you also add the
|
||||||
|
namespace podinfo.
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
apiVersion: v1
|
||||||
|
kind: Namespace
|
||||||
|
metadata:
|
||||||
|
name: podinfo
|
||||||
|
```
|
||||||
|
|
||||||
|
Then create a Kustomization to deploy podinfo.
|
||||||
|
```yaml
|
||||||
|
apiVersion: kustomize.toolkit.fluxcd.io/v1beta1
|
||||||
|
kind: Kustomization
|
||||||
|
metadata:
|
||||||
|
name: podinfo
|
||||||
|
namespace: flux-system
|
||||||
|
spec:
|
||||||
|
interval: 5m
|
||||||
|
targetNamespace: podinfo
|
||||||
|
path: ./clusters/my-cluster/podinfo
|
||||||
|
prune: true
|
||||||
|
sourceRef:
|
||||||
|
kind: GitRepository
|
||||||
|
name: flux-system
|
||||||
|
healthChecks:
|
||||||
|
- apiVersion: apps/v1
|
||||||
|
kind: Deployment
|
||||||
|
name: podinfo
|
||||||
|
namespace: podinfo
|
||||||
|
timeout: 1m
|
||||||
|
```
|
||||||
|
|
||||||
|
## Creating the Git provider
|
||||||
|
|
||||||
|
Creating a git provider is similar to creating other providers.
|
||||||
|
The only difference is that with git providers `spec.address` needs to point to the same git repository that the event source originates from.
|
||||||
|
|
||||||
|
=== "GitHub"
|
||||||
|
```yaml
|
||||||
|
apiVersion: notification.toolkit.fluxcd.io/v1beta1
|
||||||
|
kind: Provider
|
||||||
|
metadata:
|
||||||
|
name: flux-system
|
||||||
|
namespace: flux-system
|
||||||
|
spec:
|
||||||
|
type: github
|
||||||
|
address: https://github.com/<username>/fleet-infra
|
||||||
|
secretRef:
|
||||||
|
name: git
|
||||||
|
```
|
||||||
|
|
||||||
|
=== "GitLab"
|
||||||
|
```yaml
|
||||||
|
apiVersion: notification.toolkit.fluxcd.io/v1beta1
|
||||||
|
kind: Provider
|
||||||
|
metadata:
|
||||||
|
name: flux-system
|
||||||
|
namespace: flux-system
|
||||||
|
spec:
|
||||||
|
type: gitlab
|
||||||
|
address: https://gitlab.com/<username>/fleet-infra
|
||||||
|
secretRef:
|
||||||
|
name: git
|
||||||
|
```
|
||||||
|
=== "Bitbucket"
|
||||||
|
```bash
|
||||||
|
```
|
||||||
|
=== "Azure DevOps"
|
||||||
|
```bash
|
||||||
|
```
|
||||||
|
|
||||||
|
## Creating the alert
|
||||||
|
|
||||||
|
=== "GitHub"
|
||||||
|
```bash
|
||||||
|
$ flux create alert podinfo \
|
||||||
|
--provider-ref git \
|
||||||
|
--event-severity info \
|
||||||
|
--event-source 'Kustomization/podinfo' \
|
||||||
|
--namespace 'flux-system' \
|
||||||
|
--export
|
||||||
|
---
|
||||||
|
apiVersion: notification.toolkit.fluxcd.io/v1beta1
|
||||||
|
kind: Alert
|
||||||
|
metadata:
|
||||||
|
name: podinfo
|
||||||
|
namespace: flux-system
|
||||||
|
spec:
|
||||||
|
providerRef:
|
||||||
|
name: flux-system
|
||||||
|
eventSeverity: info
|
||||||
|
eventSources:
|
||||||
|
- kind: Kustomization
|
||||||
|
name: podinfo
|
||||||
|
namespace: flux-system
|
||||||
|
```
|
||||||
|
=== "GitLab"
|
||||||
|
```bash
|
||||||
|
```
|
||||||
|
=== "Bitbucket"
|
||||||
|
```bash
|
||||||
|
```
|
||||||
|
=== "Azure DevOps"
|
||||||
|
```bash
|
||||||
|
```
|
||||||
|
|
||||||
|
|
||||||
|
By now the fleet-infra repository should have a similar directory structure.
|
||||||
|
```
|
||||||
|
fleet-infra
|
||||||
|
└── clusters/
|
||||||
|
└── my-cluster/
|
||||||
|
├── flux-system/
|
||||||
|
│ ├── gotk-components.yaml
|
||||||
|
│ ├── gotk-sync.yaml
|
||||||
|
│ └── kustomization.yaml
|
||||||
|
├── podinfo/
|
||||||
|
│ ├── namespace.yaml
|
||||||
|
│ ├── deployment.yaml
|
||||||
|
│ ├── hpa.yaml
|
||||||
|
│ ├── service.yaml
|
||||||
|
│ └── kustomization.yaml
|
||||||
|
├── podinfo-kustomization.yaml
|
||||||
|
└── podinfo-notification.yaml
|
||||||
|
```
|
||||||
|
|
||||||
|
If podinfo is deployed and the health checks pass you should get a successful status in
|
||||||
|
your forked podinfo repository.
|
||||||
|
|
||||||
|
If everything is setup correctly there should now be a green check-mark next to the latest commit.
|
||||||
|
Clicking the check-mark should show a detailed view.
|
||||||
|
|
||||||
|
=== "Github"
|
||||||
|
|
||||||
|
=== "GitLab"
|
||||||
|
 |
|
||||||
|
|
||||||
|
## Generate an error
|
||||||
|
|
||||||
|
A deployment failure can be forced by setting an invalid image tag in the podinfo deployment.
|
||||||
|
```yaml
|
||||||
|
apiVersion: apps/v1
|
||||||
|
kind: Deployment
|
||||||
|
spec:
|
||||||
|
template:
|
||||||
|
spec:
|
||||||
|
containers:
|
||||||
|
- name: podinfod
|
||||||
|
image: ghcr.io/stefanprodan/podinfo:fake
|
||||||
|
```
|
||||||
|
|
||||||
|
After the commit has been reconciled it should return a failed commit status.
|
||||||
|
|
||||||
|
=== "GitHub"
|
||||||
|
|
||||||
|
=== "GitLab"
|
||||||
|
 |
|
||||||
|
|
||||||
|
|
||||||
Loading…
Reference in New Issue