Add notifications guide

docs/get-started/index.md

@@ -21,7 +21,7 @@ export GITHUB_USER=<your-username>
 
 To install the latest `tk` release run:
 
-```bash
+```sh
 curl -s https://toolkit.fluxcd.io/install.sh | sudo bash
 ```
 
@@ -58,7 +58,7 @@ kubectl cluster-info --context kind-staging
 
 Verify that your staging cluster satisfies the prerequisites with:
 
-```text
+```console
 $ tk check --pre
 
 ► checking prerequisites
@@ -104,22 +104,9 @@ $ tk bootstrap github --owner=gitopsrun --repository=fleet-infra --path=staging-
 ✚ generating manifests
 ✔ components manifests pushed
 ► installing components in gitops-system namespace
-namespace/gitops-system created
-customresourcedefinition.apiextensions.k8s.io/gitrepositories.source.fluxcd.io created
-customresourcedefinition.apiextensions.k8s.io/helmcharts.source.fluxcd.io created
-customresourcedefinition.apiextensions.k8s.io/helmrepositories.source.fluxcd.io created
-customresourcedefinition.apiextensions.k8s.io/kustomizations.kustomize.fluxcd.io created
-customresourcedefinition.apiextensions.k8s.io/profiles.kustomize.fluxcd.io created
-role.rbac.authorization.k8s.io/crd-controller-gitops-system created
-rolebinding.rbac.authorization.k8s.io/crd-controller-gitops-system created
-clusterrolebinding.rbac.authorization.k8s.io/cluster-reconciler-gitops-system created
-service/source-controller created
-deployment.apps/kustomize-controller created
-deployment.apps/source-controller created
-networkpolicy.networking.k8s.io/deny-ingress created
-Waiting for deployment "source-controller" rollout to finish: 0 of 1 updated replicas are available...
 deployment "source-controller" successfully rolled out
 deployment "kustomize-controller" successfully rolled out
+deployment "notification-controller" successfully rolled out
 ✔ install completed
 ► configuring deploy key
 ✔ deploy key configured
@@ -208,7 +195,7 @@ git add -A && git commit -m "add staging webapp" && git push
 
 In about 30s the synchronization should start:
 
-```text
+```console
 $ watch tk get kustomizations
 
 ✔ gitops-system last applied revision master/35d5765a1acb9e9ce66cad7274c6fe03eee1e8eb
@@ -219,7 +206,7 @@ $ watch tk get kustomizations
 
 When the synchronization finishes you can check that the webapp services are running:
 
-```text
+```console
 $ kubectl -n webapp get deployments,services
 
 NAME                       READY   UP-TO-DATE   AVAILABLE   AGE
@@ -307,7 +294,7 @@ git add -A && git commit -m "add prod webapp" && git push
 
 List git sources:
 
-```text
+```console
 $ tk get sources git
 
 ✔ gitops-system last fetched revision master/99072ee132abdead8b7799d7891eae2f524eb73d
@@ -318,7 +305,7 @@ The kubectl equivalent is `kubectl -n gitops-system get gitrepositories`.
 
 List kustomization:
 
-```text
+```console
 $ tk get kustomizations
 
 ✔ gitops-system last applied revision master/99072ee132abdead8b7799d7891eae2f524eb73d
@@ -341,7 +328,7 @@ git add -A && git commit -m "update prod webapp" && git push
 
 Trigger a git sync:
 
-```text
+```console
 $ tk sync ks gitops-system --with-source 
 
 ► annotating source gitops-system
@@ -358,7 +345,7 @@ The kubectl equivalent is `kubectl -n gitops-system annotate gitrepository/gitop
 
 Wait for the webapp to be upgraded:
 
-```text
+```console
 $ watch tk get kustomizations
 
 ✔ gitops-system last applied revision master/d751ea264d48bf0db8b588d1d08184834ac8fec9

docs/guides/notifications.md

@@ -0,0 +1,101 @@
+# Setup Notifications
+
+When operating a cluster, different teams may wish to receive notifications about
+the status of their GitOps pipelines.
+For example, the on-call team would receive alerts about reconciliation
+failures in the cluster, while the dev team may wish to be alerted when a new version 
+of an app was deployed and if the deployment is healthy.
+
+## Prerequisites
+
+* [Get started guide](../get-started/index.md)
+
+The GitOps toolkit controllers emit Kubernetes events whenever a resource status changes.
+You can use the [notification-controller](../components/notification/controller.md)
+to forward these events to Slack, Microsoft Teams, Discord or Rocket chart.
+The notification controller is part of the default toolkit installation.
+
+## Define a provider
+
+First create a secret with your Slack incoming webhook:
+
+```sh
+kubectl -n gitops-system create secret generic slack-url \
+--from-literal=address=https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK
+```
+
+Note that the secret must contain an `address` field,
+it can be a Slack, Microsoft Teams, Discord or Rocket webhook URL.
+
+Create a notification provider for Slack by referencing the above secret:
+
+```yaml
+apiVersion: notification.fluxcd.io/v1alpha1
+kind: Provider
+metadata:
+  name: slack
+  namespace: gitops-system
+spec:
+  type: slack
+  channel: general
+  secretRef:
+    name: slack-url
+```
+
+The provider type can be `slack`, `msteams`, `discord`, `rocket` or `webhook`.
+
+When type `webhook` is specified, the notification controller will post the incoming
+[event](../components/notification/event.md) in JSON format to the webhook address. 
+This way you can create custom handlers that can store the events in 
+Elasticsearch, CloudWatch, Stackdriver, etc.
+
+## Define an alert
+
+Create an alert definition for the webapp kustomizations:
+
+```yaml
+apiVersion: notification.fluxcd.io/v1alpha1
+kind: Alert
+metadata:
+  name: on-call-webapp
+  namespace: gitops-system
+spec:
+  providerRef: 
+    name: slack
+  eventSeverity: info
+  eventSources:
+    - kind: Kustomization
+      name: webapp-backend
+    - kind: Kustomization
+      name: webapp-frontend
+```
+
+Apply the above files or commit them to the `fleet-infra` repository.
+
+To verify that the alert has been acknowledge by the notification controller do:
+
+```console
+$ kubectl -n gitops-system get alerts
+
+NAME             READY   STATUS        AGE
+on-call-webapp   True    Initialized   1m
+```
+
+Multiple alerts can be used to send notifications to different channels or Slack organizations.
+
+The event severity can be set to `info` or `error`.
+When the severity is set to `error`, the kustomize controller will alert on any error
+encountered during the reconciliation process.
+This includes kustomize build and validation errors,
+apply errors and health check failures.
+
+![error alert](../diagrams/slack-error-alert.png)
+
+When the verbosity is set to `info`, the controller will alert if:
+
+* a Kubernetes object was created, updated or deleted
+* heath checks are passing
+* a dependency is delaying the execution
+* an error occurs
+
+![info alert](../diagrams/slack-info-alert.png)