From 15f8e6369bd3053eb3fd6c3e5176907b85a55ff8 Mon Sep 17 00:00:00 2001 From: Michael Bridgen Date: Wed, 27 Jan 2021 18:38:49 +0000 Subject: [PATCH 01/13] Add image automation migration how-to This doc describes how to move from using Flux v1 to update image refs in git, to using Flux v2. There is some overlap with the tutorial on how to use Flux v2 automation. This how-to spends more time on how to convert existing configuration to be used with Flux v2. Signed-off-by: Michael Bridgen --- docs/guides/flux-v1-automation-migration.md | 723 ++++++++++++++++++++ 1 file changed, 723 insertions(+) create mode 100644 docs/guides/flux-v1-automation-migration.md diff --git a/docs/guides/flux-v1-automation-migration.md b/docs/guides/flux-v1-automation-migration.md new file mode 100644 index 00000000..a6c2a864 --- /dev/null +++ b/docs/guides/flux-v1-automation-migration.md @@ -0,0 +1,723 @@ + +# Migrating image update automation to Flux v2 + +"Image Update Automation" is a process in which Flux makes commits to your Git repository when it +detects that there is a new image to be used in a workload (e.g., a Deployment). In Flux v2 this +works quite differently to how it worked in Flux v1. This guide explains the differences and how to +port your cluster configuration from v1 to v2. There is also a [tutorial for using image update +automation with a new cluster][image-update-tute]. + +## Overview of changes between v1 and v2 + +In Flux v1, image update automation (from here, just "automation") was built into the Flux daemon, +which scanned everything it found in the cluster and updated the Git repository it was syncing. + +In Flux v2, + + - automation is controlled with custom resources, not annotations + - ordering images by build time is not supported (there is [a section + below](#how-to-migrate-annotations-to-image-policies) explaining what to do instead) + - the fields to update in files are marked explicitly, rather than inferred from annotations. + +#### Automation is now controlled by custom resources + +Flux v2 breaks down the functions in Flux v1's daemon into controllers, with each having a specific +area of concern. Automation is now done by two controllers: one which scans image repositories to +find the latest images, and one which uses that information to commit changes to git +repositories. These are in turn separate to the syncing controllers. + +This means that automation in Flux v2 is governed by custom resources. In Flux v1 the daemon scanned +everything, and looked at annotations on the resources to determine what to update. Automation in v2 +is more explicit than in v1 -- you have to mention exactly which images you want to be scanned, and +which fields you want to be updated. + +A consequence of using custom resources is that with Flux v2 you can have an arbitrary number of +automations, targeting different Git repositories if you wish, and updating different sets of +images. If you run a multitenant cluster, the tenants can define automation in their own namespaces, +for their own Git repositories. + +#### Selecting an image is more flexible + +The ways in which you choose to select an image have changed. In Flux v1, you generally supply a +filter pattern, and the latest image is the image with the most recent build time out of those +filtered. In Flux v2, you choose an ordering, and separately specify a filter for the tags to +consider. These are dealt with in detail below. + +Selecting an image by build time is no longer supported. This is the implicit default in Flux v1. In +Flux v2, you will need to tag images so that they sort in the order you would like -- [see +below](#how-to-use-timestamps-in-image-tags) for how to do this conveniently. + +#### Fields to update are explicitly marked + +Lastly, in Flux v2 the fields to update in files are marked explicitly. In Flux v1 they are inferred +from the type of the resource, along with the annotations given. The approach in Flux v1 was limited +to the types that had been programmed in, whereas Flux v2 can update any Kubernetes object (and some +files that aren't Kubernetes objects, like `kustomization.yaml`). + +## Preparing for migration + +It is best to complete migration of your system to _Flux v2 syncing_ first, using the [Flux v1 +migration guide][flux-v1-migration]. This will remove Flux v1 from the system, along with its image +automation. You can then reintroduce automation with Flux v2 by following the instructions in this +guide. + +It is safe to leave the annotations for Flux v1 in files while you reintroduce automation, because +Flux v2 will ignore them. + +To migrate to Flux v2 automation, you will need to do three things: + + - make sure you are running the automation controllers; and, + - translate Flux v1 annotations to Flux v2 `ImageRepository` and `ImagePolicy` objects, and put + update markers in files; and, + - declare the automation with an `ImageUpdateAutomation` object. + +A Flux v2 installation will typically have a Git repository structured like this: + +``` +<...>/flux-system/ + gotk-components.yaml + gotk-sync.yaml +<...>/app/ + # deployments etc. +``` + +The `<...>` is the path to a particular cluster's definitions -- this may be simply `.`, if you did +not supply a `--path` flag when bootstrapping the first time, or something like +`clusters/my-cluster`. To get the files in the right place, set a variable for this path: + +```bash +$ CLUSTER_PATH=<...> # e.g., "." or "clusters/my-cluster", or ... +$ AUTO_PATH=$CLUSTER_PATH/automation +$ mkdir ./$AUTO_PATH +``` + +The file `$CLUSTER_PATH/flux-system/gotk-components.yaml` has definitions of all the Flux v2 +controllers and custom resource definitions. The file `gotk-sync.yaml` defines a `GitRepository` and +a `Kustomization` which will sync the Git repository itself. + +To these will be added definitions for automation objects. This guide puts manifest files for +automation in `$CLUSTER_PATH/automation/`, but there is no particular structure required +by Flux. The automation objects do not have to be in the same namespace as the objects to be +updated. + +The command-line tool `flux` will be used below; see [these instructions][install-cli] for how to +install it. + +## Running the automation controllers + +The first thing to do is to deploy the automation controllers to your cluster. The best way to +proceed will depend on the approach you took when following the [Flux read-only migration +guide][flux-v1-migration]. + + - If you used `flux bootstrap` to create a new Git repository, then ported your cluster + configuration to that repository, use [After `flux bootstrap`](#after-flux-bootstrap); + - If you used `flux install` to install the controllers directly, use [After migrating Flux v1 in + place](#after-migrating-flux-v1-in-place); + - If you used `flux install` and exported the configuration to a file, use [After committing Flux + v2 configuration to Git](#after-committing-flux-v2-configuration-to-git). + +### After `flux bootstrap` + +When starting from scratch, you are likely to have used `flux bootstrap`. Rerun the command, and +include the image automation controllers in your starting configuration with the flag +`--components-extra`, [as shown in the installation guide][flux-bootstrap]. + +This will commit changes to your Git repository and sync them in the cluster. + +```bash +flux check --components-extra=image-reflector-controller,image-automation-controller +``` + +Now jump to the section [Migrating each manifest to Flux v2](#migrating-each-manifest-to-flux-v2). + +### After migrating Flux v1 in place + +If you followed the [Flux v1 migration guide][flux-v1-migration], you will already be running some +Flux v2 controllers. The automation controllers are currently considered an optional extra to those, +but are installed and run in much the same way. You may or may not have committed the Flux v2 +configuration to your Git repository. If you did, go to the section [After committing Flux v2 +configuration to Git](#after-committing-flux-v2-configuration-to-git). + +If _not_, you will be installing directly to the cluster: + +```bash +$ flux install --components-extra=image-reflector-controller,image-automation-controller +``` + +It is safe to repeat the installation command, or to run it after using `flux bootstrap`, so long as +you repeat any arguments you supplied the first time. + +Now jump ahead to [Migrating each manifest to Flux v2](#migrating-each-manifest-to-flux-v2). + +#### After committing a Flux v2 configuration to Git + +If you added the Flux v2 configuration to your git repository, assuming it's in the file +`$CLUSTER_PATH/flux-system/gotk-components.yaml` as used in the guide, use `flux install` and write +it back to that file: + +```bash +$ flux install \ + --components-extra=image-reflector-controller,image-automation-controller \ + --export > "$CLUSTER_PATH/flux-system/gotk-components.yaml" +``` + +Commit changes to the `$CLUSTER_PATH/flux-system/gotk-components.yaml` file and sync the cluster: + +```bash +$ git add $CLUSTER_PATH/flux-system/gotk-components.yaml +$ git commit -s -m "Add image automation controllers to Flux config" +$ git push +$ flux reconcile kustomization --with-source flux-system +``` + +Read on to the next section to see how to change each manifest file to work with Flux v2. + +## Migrating each manifest to Flux v2 + +In Flux v1, the annotation + + fluxcd.io/automated: "true" + +switches automation on for a manifest (a description of a Kubernetes object). For each manifest that +has that annotation, you will need to create custom resources to scan for the latest image, and to +replace the annotations with field markers. + +The following sections explain these steps, using this example Deployment manifest which is +initially annotated to work with Flux v1: + +```yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: my-app + namespace: default + annotations: + fluxcd.io/automated: "true" + fluxcd.io/tag.app: semver:^5.0 + selector: + matchLabels: + app: podinfo +spec: + template: + metadata: + labels: + app: podinfo + spec: + containers: + - name: app + image: ghcr.io/stefanprodan/podinfo:5.0.0 +``` + +!!! warning + A YAML file may have more than one manifest in it, separated with + `---`. Be careful to account for each manifest in a file. + +You may wish to try migrating the automation of just one file or manifest and follow it through to +the end of the guide, before returning here to do the remainder. + +### How to migrate annotations to image policies + +For each image repository that is the subject of automation you will need to create an +`ImageRepository` object, so that the image repository is scanned for tags. The image repository in +the example deployment is `ghcr.io/stefanprodan/podinfo`, which is the image reference minus its +tag: + +```yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: my-app + namespace: default + annotations: + fluxcd.io/automated: "true" + fluxcd.io/tag.app: semver:^5.0 + selector: + matchLabels: + app: podinfo +spec: + template: + metadata: + labels: + app: podinfo + spec: + containers: + - name: app + image: ghcr.io/stefanprodan/podinfo:5.0.0 # <-- image reference +``` + +The command-line tool `flux` will help create a manifest for you. Note that the output is redirected +to a file under `$AUTO_PATH`, so it can be added to the Git repository and synced to the cluster. + +```bash +$ # the environment variable $AUTO_PATH was set earlier +$ flux create image repository podinfo-image \ + --image ghcr.io/stefanprodan/podinfo \ + --interval 5m \ + --export > ./$AUTO_PATH/podinfo-image.yaml +$ cat ./$AUTO_PATH/podinfo-image.yaml +--- +apiVersion: image.toolkit.fluxcd.io/v1alpha1 +kind: ImageRepository +metadata: + name: podinfo-image + namespace: flux-system +spec: + image: ghcr.io/stefanprodan/podinfo + interval: 5m0s +``` + +!!! hint + If you are using the same image repository in several manifests, you only need one + `ImageRepository` object for it. + +##### Using image registry credentials for scanning + +When your image repositories are private, you supply Kubernetes with "image pull secrets" with +credentials for accessing the image registry (e.g., DockerHub). The image reflector controller needs +the same kind of credentials to scan image repositories. + +There are several ways that image pull secrets can be made available for the image reflector +controller. The [image update tutorial][image-update-tute-creds] describes how to create or arrange +secrets for scanning to use. Also see later in the tutorial for [instructions specific to some cloud +platforms][image-update-tute-clouds]. + +##### Committing and checking the ImageRepository + +%%% TODO commit, push, reconcile, check the status of the ImageRepository + +#### Replacing automation annotations + +For each _field_ that's being updated by automation, you'll need an `ImagePolicy` object to describe +how to select an image for the field value. In the example, the field `.image` in the container +named `"app"` is the field being updated. + +In Flux v1, annotations describe how to select the image to update to, using a prefix. In the +example, the prefix is `semver:`: + +```yaml + annotations: + fluxcd.io/automated: "true" + fluxcd.io/tag.app: semver:^5.0 +``` + +These are the prefixes supported in Flux v1: + +| Flux v1 prefix | Meaning | +|------------------|-------------| +| `glob:` | Filter for tags matching the glob pattern, then select the newest by build time | +| `regex:` | Filter for tags matching the regular expression, then select the newest by build time | +| `semver:` | Filter for tags that represent versions, and select the highest version in the given range | + +Flux v2 does not support selecting the lastest image by build time, because that requires the +container config file for each image to be fetched, and this operation is subject to strict rate +limiting by image registries (e.g., by [DockerHub][dockerhub-rates]). The suggested way to select by +image build time is to put a timestamp in each image tag, as explained below. + +If you are using ... + +| Flux v1 prefix | Flux v2 equivalent | +|-----------------|--------------------| +| glob: | [Use timestamped tags](#how-to-use-timestamps-in-image-tags) | +| regex: | [Use timestamped tags](#how-to-use-timestamp-in-image-tags) | +| semver: | [Use semver ordering](#how-to-use-semver-image-tags) | + +#### How to use timestamps in image tags + +To give image tags a useful ordering, you can use a timestamp as part of each image's tag, then sort +alphabetically. + +This is a change from Flux v1, in which the build time was fetched from each image's config, and +didn't need to be included in the image tag. Therefore, this is likely to require a change to your +build process. + +##### Example of a build process with timestamp tagging + +Here is an example of a [GitHub Actions job][gha-syntax] that creates a "build ID" with the git +branch, SHA1, and a timestamp, and uses it as a tag when building an image: + +```yaml +jobs: + build-push: + env: + IMAGE: org/my-app + runs-on: ubuntu-latest + steps: + + - name: Generate build ID + id: prep + run: | + branch=${GITHUB_REF##*/} + sha=${GITHUB_SHA::8} + ts=$(date +%s) + echo "::set-output name=BUILD_ID::${branch}-${sha}-${ts}" + + # These are prerequisites for the docker build step + - name: Set up QEMU + uses: docker/setup-qemu-action@v1 + - name: Set up Docker Buildx + uses: docker/setup-buildx-action@v1 + - name: Login to DockerHub + uses: docker/login-action@v1 + with: + username: ${{ secrets.DOCKERHUB_USERNAME }} + password: ${{ secrets.DOCKERHUB_TOKEN }} + + - name: Publish multi-arch container image + uses: docker/build-push-action@v2 + with: + push: true + context: . + file: ./Dockerfile + tags: | + ${{ env.IMAGE }}:${{ steps.prep.outputs.BUILD_ID }} +``` + +##### Formats and alternatives + +The important properties for sorting alphabetically to work well are that the parts of the timestamp +go from most significant to least (e.g., the year down to the second), and that the output is always +the same number of characters. + +Image tags often show in user interfaces, so readability matters. Here are some alternatives: + +```bash +$ # seconds-since-epoch (used in the example above) +$ date +%s +1611840548 +$ # date and time (remember ':' is not allowed in a tag) +$ date +%F.%H%M%S +2021-01-28.133158 +``` + +Alternatively, you can use a stable serial number as part of the tag. Some CI platforms will +provide a build number in an environment variable, but that may not be reliable to use as a serial +number -- check the platform documentation. + +A commit count can be a reasonable stand-in for a serial number, if you build an image per commit, +and you don't rewrite the branch in question. + +```bash +$ # commits in branch +$ git --rev-list --count HEAD +1504 +``` + +Beware: this will not give a useful number if you have a shallow clone. + +##### Filtering the tags in an `ImagePolicy` + +The timestamp (or serial number) is the part of the tag that you want to order on. The SHA1 is there +so you can track an image back to the commit from which it was built. You don't need the branch for +sorting, but you may want to include only builds from a specific branch. + +Say you want to filter for only images that are from `main` branch, and pick the most recent. Your +`ImagePolicy` would look like this: + +```yaml +apiVersion: image.toolkit.fluxcd.io/v1alpha1 +kind: ImagePolicy +metadata: + name: my-app-policy + namespace: flux-system +spec: + imageRepositoryRef: + name: podinfo-image + filterTags: + pattern: '^main-[a-f0-9]+-(?P[0-9]+)' + extract: '$ts' + policy: + alphabetical: + order: asc +``` + +The `.spec.pattern` field gives a regular expression that a tag must match to be included. The +`.spec.extract` field gives a replacement pattern that can refer back to capture groups in the +filter pattern. The extracted values are sorted to find the selected image tag. In this case, the +timestamp part of the tag will be extracted and sorted alphabetically in ascending order. See [the +reference docs][imagepolicy-ref] for more examples. + +Once you have made sure you have image tags and an `ImagePolicy` that works, jump ahead to [Checking +the ImagePolicy works](#checking-the-image-policy-works). + +### How to use SemVer image tags + +The other kind of sorting is by [SemVer][semver], picking the highest version from among those +included by the filter. A semver range will also filter for tags that fit in the range. For example, + +```yaml + semver: + range: ^5.0 +``` + +includes only tags that have a major version of `5`, and selects whichever is the highest. + +This can be combined with a regular expression pattern, to filter on other parts of the tags. For +example, you might put a target environment as well as the version in your image tags, like +`dev-v1.0.3`. + +Then you would use an `ImagePolicy` similar to this one: + +```yaml +apiVersion: image.toolkit.fluxcd.io/v1alpha1 +kind: ImagePolicy +metadata: + name: my-app-policy + namespace: flux-system +spec: + imageRepositoryRef: + name: podinfo-image + filterTags: + pattern: '^dev-v(?P.*)' + extract: '$version' + policy: + semver: + range: '^1.0' +``` + +Continue on to the next section to check that your `ImagePolicy` works. + +### Checking the ImagePolicy works + +%%% TODO commit, push, reconcile, check the status of the ImagePolicy + +#### An ImagePolicy for the example + +The example Deployment has annotations using `semver:` as a prefix, so the policy object also uses +semver: + +```bash +$ # the environment variable $AUTO_PATH was set earlier +$ flux create image policy my-app-policy \ + --image-ref podinfo-image \ + --semver '^5.0' \ + --export > ./$AUTO_PATH/my-app-policy.yaml +$ cat ./$AUTO_PATH/my-app-policy.yaml +--- +apiVersion: image.toolkit.fluxcd.io/v1alpha1 +kind: ImagePolicy +metadata: + name: my-app-policy + namespace: flux-system +spec: + imageRepositoryRef: + name: podinfo-image + policy: + semver: + range: ^5.0 +``` + +### How to mark up files for update + +The last thing to do in each manifest is to mark the fields that you want to be updated. + +In Flux v1, the annotations in a manifest determines the fields to be updated. In the example, the +annotations target the image used by the container `app`: + +```yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: my-app + namespace: default + annotations: + fluxcd.io/automated: "true" + fluxcd.io/tag.app: semver:^5.0 # <-- `.app` here + selector: + matchLabels: + app: podinfo +spec: + template: + metadata: + labels: + app: podinfo + spec: + containers: + - name: app # <-- targets `app` here + image: ghcr.io/stefanprodan/podinfo:5.0.0 +``` + +This works straight-forwardly for Deployment manifests, but when it comes to `HelmRelease` +manifests, it [gets complicated][helm-auto], and it doesn't work at all for many kinds of resources. + +For Flux v2, you mark the field you want to be updated directly, with the namespaced name of the +image policy to apply. This is the example Deployment, marked up for Flux v2: + +```yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + namespace: default + name: my-app + selector: + matchLabels: + app: podinfo +spec: + template: + metadata: + labels: + app: podinfo + spec: + containers: + - name: app + image: ghcr.io/stefanprodan/podinfo:5.0.0 # {"$imagepolicy": "flux-system:my-app-policy"} +``` + +The value `flux-system:my-app-policy` names the policy that selects the desired image. + +This works in the same way for `DaemonSet` and `CronJob` manifests. For `HelmRelease` manifests, put +the marker alongside the part of the `values` that has the image tag. If the image tag is a separate +field, you can put `:tag` on the end of the name, to replace the value with just the selected +image's tag. The [image automation guide][image-auto-guide] has examples for `HelmRelease` and other +custom resources. + +## Controlling automation + +In Flux v1, automation was run by default. With Flux v2, you have to explicitly tell the controller +which Git repository to update and how to do so. These are defined in an `ImageUpdateAutomation` +object; but first, you need a `GitRepository` with write access, for the automation to use. + +If you followed the [Flux v1 read-only migration guide][flux-v1-migration], you will have a +`GitRepository` defined in the namespace `flux-system`, for syncing to use. This `GitRepository` +will have _read_ access to the Git repository by default, and automation needs _write_ access to +push commits. + +To give it write access, you can replace the secret it refers to. How to do this will depend on what +kind of authentication you used to install Flux v2. + +### Replacing the Git credentials secret + +The secret with Git credentials will be named in the `.spec.secretRef.name` field of the +`GitRepository` object. Say your `GitRepository` is in the _namespace_ `flux-system` and _named_ +`flux-system` (these are the defaults if you used `flux bootstrap`); you can retrieve the secret +name and Git URL with: + +```bash +$ FLUX_NS=flux-system +$ GIT_NAME=flux-system +$ SECRET_NAME=$(kubectl -n $FLUX_NS get gitrepository $GIT_NAME -o jsonpath={.spec.secretRef.name}) +$ GIT_URL=$(kubectl -n $FLUX_NS get gitrepository $GIT_NAME -o jsonpath='{.spec.url}') +$ echo $SECRET_NAME $GIT_URL # make sure they have values +``` + +If you're not sure which kind of credentials you're using, look at the secret: + +```bash +$ kubectl -n $FLUX_NS describe secret $SECRET_NAME +``` + +An entry at `.data.identity` indicates that you are using an SSH key (the [first +section](#replacing-an-ssh-key-secret) below); an entry at `.data.username` indicates you are using +a username and password or token (the [second section](#replacing-a-username-password-secret) +below). + +#### Replacing an SSH key secret + +When using an SSH (deploy) key, create a new key: + +```bash +$ flux create secret git -n $FLUX_NS $SECRET_NAME --url=$GIT_URL +``` + +You will need to copy the public key that's printed out, and install that as a deploy key for your +Git repo **making sure to check the 'All write access' box** (or otherwise give the key write +permissions). Remove the old deploy key. + +#### Replacing a username/password secret + +When you're using a username and password to authenticate, you may be able to change the permissions +associated with that account. + +If not, you will need to create a new access token (e.g., ["Personal Access Token"][github-pat] in +GitHub). In this case, once you have the new token you can replace the secret with the following: + +```bash +$ flux create secret -n $FLUX_NS git $GIT_NAME \ + --username --password --url $GIT_URL +``` + +#### Checking the new credentials + +To check if your replaced credentials still work, try syncing the `GitRepository` object: + +```bash +$ flux reconcile source git -n $FLUX_NS $GIT_NAME +► annotating GitRepository flux-system in flux-system namespace +✔ GitRepository annotated +◎ waiting for GitRepository reconciliation +✔ GitRepository reconciliation completed +✔ fetched revision main/d537304e8f5f41f1584ca1e807df5b5752b2577e +``` + +Assuming this is successful, it tells you the new credentials have at least read access. + +### Making an automation object + +To set automation running, you create an [`ImageUpdateAutomation`][auto-ref] object. Each object +will update a Git repository according to the image policy objects in the object's namespace. + +Here is an `ImageUpdateAutomation` manifest for the example (note: you will have to supply your own +value for at least the host part of the email address): + +```yaml +$ FLUXBOT_EMAIL=fluxbot@example.com # supply your own host or address here +$ flux create image update my-app-auto \ + --author-name FluxBot --author-email "$FLUXBOT_EMAIL" \ + --git-repo-ref $GIT_NAME --branch main \ + --interval 5m \ + --export > my-app-auto.yaml +$ cat my-app-auto.yaml +--- +apiVersion: image.toolkit.fluxcd.io/v1alpha1 +kind: ImageUpdateAutomation +metadata: + name: my-app-auto + namespace: flux-system +spec: + checkout: + branch: main + gitRepositoryRef: + name: flux-system + commit: + authorEmail: fluxbot@example.com + authorName: FluxBot + interval: 5m0s +``` + +#### Commit and check that the automation object works + +%%% TODO commit, push, reconcile, then check status, then check that it created a commit (if it was going to ..) + +# %%% TODO %%% + +**Actually create the objects -- sync them from git?** + +**Encrypting image pull / certificate secrets** + +**What to look at to see if it's working** + +**Where to put repositories and policies (commit them to git!)** + +**What to do if you run a build step which loses comments** + +**How to lock / remove / pause automation** + +**Is there an equivalent to the 'ignore' annotation?** + +**Terraform installation?** + +**The Flux v1 migration guide has a branch where you don't put gotk-components.yaml in the repo** + +[image-update-tute]: https://toolkit.fluxcd.io/guides/image-update/ +[dockerhub-rates]: https://docs.docker.com/docker-hub/billing/faq/#pull-rate-limiting-faqs +[gha-syntax]: https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions +[imagepolicy-ref]: https://toolkit.fluxcd.io/components/image/imagepolicies/ +[helm-auto]: https://docs.fluxcd.io/en/1.21.1/references/helm-operator-integration/#automated-image-detection). +[image-auto-guide]: https://toolkit.fluxcd.io/guides/image-update/#configure-image-update-for-custom-resources +[flux-v1-migration]: ../flux-v1-migration/ +[install-cli]: https://toolkit.fluxcd.io/get-started/#install-the-flux-cli +[image-auto-guide]: ../image-update/ +[flux-bootstrap]: https://toolkit.fluxcd.io/guides/installation/#bootstrap +[github-pat]: https://docs.github.com/en/github/authenticating-to-github/creating-a-personal-access-token +[auto-object-ref]: https://toolkit.fluxcd.io/components/image/imageupdateautomations/ +[image-update-tute-creds]: https://toolkit.fluxcd.io/guides/image-update/#configure-image-scanning +[image-update-tute-clouds]: https://toolkit.fluxcd.io/guides/image-update/#imagerepository-cloud-providers-authenticatio From 2a3a4456c19b891cdb03f94fa64c34cc4b6ed7a2 Mon Sep 17 00:00:00 2001 From: Michael Bridgen Date: Mon, 8 Feb 2021 12:34:47 +0000 Subject: [PATCH 02/13] Separate image tags howto from migration howto The Flux v1 migration how-to flows better if the section on how to set builds up to tag images in the right way is its own document. It's a lot to skim past when you don't need it, and (since it's a different layer of yak hair) something you might want to figure out first if you do need it. Signed-off-by: Michael Bridgen --- docs/guides/flux-v1-automation-migration.md | 115 +++------------ docs/guides/sortable-image-tags.md | 150 ++++++++++++++++++++ 2 files changed, 167 insertions(+), 98 deletions(-) create mode 100644 docs/guides/sortable-image-tags.md diff --git a/docs/guides/flux-v1-automation-migration.md b/docs/guides/flux-v1-automation-migration.md index a6c2a864..a2e2242a 100644 --- a/docs/guides/flux-v1-automation-migration.md +++ b/docs/guides/flux-v1-automation-migration.md @@ -300,26 +300,13 @@ example, the prefix is `semver:`: fluxcd.io/tag.app: semver:^5.0 ``` -These are the prefixes supported in Flux v1: +These are the prefixes supported in Flux v1, and what to use in Flux v2: -| Flux v1 prefix | Meaning | -|------------------|-------------| -| `glob:` | Filter for tags matching the glob pattern, then select the newest by build time | -| `regex:` | Filter for tags matching the regular expression, then select the newest by build time | -| `semver:` | Filter for tags that represent versions, and select the highest version in the given range | - -Flux v2 does not support selecting the lastest image by build time, because that requires the -container config file for each image to be fetched, and this operation is subject to strict rate -limiting by image registries (e.g., by [DockerHub][dockerhub-rates]). The suggested way to select by -image build time is to put a timestamp in each image tag, as explained below. - -If you are using ... - -| Flux v1 prefix | Flux v2 equivalent | -|-----------------|--------------------| -| glob: | [Use timestamped tags](#how-to-use-timestamps-in-image-tags) | -| regex: | [Use timestamped tags](#how-to-use-timestamp-in-image-tags) | -| semver: | [Use semver ordering](#how-to-use-semver-image-tags) | +| Flux v1 prefix | Meaning | Flux v2 equivalent | +|------------------|-------------|--------------------| +| `glob:` | Filter for tags matching the glob pattern, then select the newest by build time | [Use timestamped tags](#how-to-use-timestamps-in-image-tags) | +| `regex:` | Filter for tags matching the regular expression, then select the newest by build time |[Use timestamped tags](#how-to-use-timestamp-in-image-tags) | +| `semver:` | Filter for tags that represent versions, and select the highest version in the given range | [Use semver ordering](#how-to-use-semver-image-tags) | #### How to use timestamps in image tags @@ -330,84 +317,18 @@ This is a change from Flux v1, in which the build time was fetched from each ima didn't need to be included in the image tag. Therefore, this is likely to require a change to your build process. -##### Example of a build process with timestamp tagging - -Here is an example of a [GitHub Actions job][gha-syntax] that creates a "build ID" with the git -branch, SHA1, and a timestamp, and uses it as a tag when building an image: - -```yaml -jobs: - build-push: - env: - IMAGE: org/my-app - runs-on: ubuntu-latest - steps: - - - name: Generate build ID - id: prep - run: | - branch=${GITHUB_REF##*/} - sha=${GITHUB_SHA::8} - ts=$(date +%s) - echo "::set-output name=BUILD_ID::${branch}-${sha}-${ts}" - - # These are prerequisites for the docker build step - - name: Set up QEMU - uses: docker/setup-qemu-action@v1 - - name: Set up Docker Buildx - uses: docker/setup-buildx-action@v1 - - name: Login to DockerHub - uses: docker/login-action@v1 - with: - username: ${{ secrets.DOCKERHUB_USERNAME }} - password: ${{ secrets.DOCKERHUB_TOKEN }} - - - name: Publish multi-arch container image - uses: docker/build-push-action@v2 - with: - push: true - context: . - file: ./Dockerfile - tags: | - ${{ env.IMAGE }}:${{ steps.prep.outputs.BUILD_ID }} -``` - -##### Formats and alternatives - -The important properties for sorting alphabetically to work well are that the parts of the timestamp -go from most significant to least (e.g., the year down to the second), and that the output is always -the same number of characters. - -Image tags often show in user interfaces, so readability matters. Here are some alternatives: +The guide [How to make sortable image tags][image-tags-guide] explains how to change your build +process to tag images with a timestamp. This will mean Flux v2 can sort the tags to find the most +recently built image. -```bash -$ # seconds-since-epoch (used in the example above) -$ date +%s -1611840548 -$ # date and time (remember ':' is not allowed in a tag) -$ date +%F.%H%M%S -2021-01-28.133158 -``` - -Alternatively, you can use a stable serial number as part of the tag. Some CI platforms will -provide a build number in an environment variable, but that may not be reliable to use as a serial -number -- check the platform documentation. - -A commit count can be a reasonable stand-in for a serial number, if you build an image per commit, -and you don't rewrite the branch in question. - -```bash -$ # commits in branch -$ git --rev-list --count HEAD -1504 -``` +##### Filtering the tags in an `ImagePolicy` -Beware: this will not give a useful number if you have a shallow clone. +The recommended format for image tags using a timestamp is: -##### Filtering the tags in an `ImagePolicy` + -- The timestamp (or serial number) is the part of the tag that you want to order on. The SHA1 is there -so you can track an image back to the commit from which it was built. You don't need the branch for +so you can trace an image back to the commit from which it was built. You don't need the branch for sorting, but you may want to include only builds from a specific branch. Say you want to filter for only images that are from `main` branch, and pick the most recent. Your @@ -567,8 +488,8 @@ The value `flux-system:my-app-policy` names the policy that selects the desired This works in the same way for `DaemonSet` and `CronJob` manifests. For `HelmRelease` manifests, put the marker alongside the part of the `values` that has the image tag. If the image tag is a separate field, you can put `:tag` on the end of the name, to replace the value with just the selected -image's tag. The [image automation guide][image-auto-guide] has examples for `HelmRelease` and other -custom resources. +image's tag. The [image automation guide][image-update-tute-custom] has examples for `HelmRelease` +and other custom resources. ## Controlling automation @@ -708,16 +629,14 @@ spec: **The Flux v1 migration guide has a branch where you don't put gotk-components.yaml in the repo** [image-update-tute]: https://toolkit.fluxcd.io/guides/image-update/ -[dockerhub-rates]: https://docs.docker.com/docker-hub/billing/faq/#pull-rate-limiting-faqs -[gha-syntax]: https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions [imagepolicy-ref]: https://toolkit.fluxcd.io/components/image/imagepolicies/ [helm-auto]: https://docs.fluxcd.io/en/1.21.1/references/helm-operator-integration/#automated-image-detection). -[image-auto-guide]: https://toolkit.fluxcd.io/guides/image-update/#configure-image-update-for-custom-resources +[image-update-tute-custom]: https://toolkit.fluxcd.io/guides/image-update/#configure-image-update-for-custom-resources [flux-v1-migration]: ../flux-v1-migration/ [install-cli]: https://toolkit.fluxcd.io/get-started/#install-the-flux-cli -[image-auto-guide]: ../image-update/ [flux-bootstrap]: https://toolkit.fluxcd.io/guides/installation/#bootstrap [github-pat]: https://docs.github.com/en/github/authenticating-to-github/creating-a-personal-access-token [auto-object-ref]: https://toolkit.fluxcd.io/components/image/imageupdateautomations/ [image-update-tute-creds]: https://toolkit.fluxcd.io/guides/image-update/#configure-image-scanning [image-update-tute-clouds]: https://toolkit.fluxcd.io/guides/image-update/#imagerepository-cloud-providers-authenticatio +[image-tags-guide]: ../sortable-image-tags/ diff --git a/docs/guides/sortable-image-tags.md b/docs/guides/sortable-image-tags.md new file mode 100644 index 00000000..72bcb2f6 --- /dev/null +++ b/docs/guides/sortable-image-tags.md @@ -0,0 +1,150 @@ + +# How to make sortable image tags to use with automation + +Flux v2 does not support selecting the lastest image by build time. Obtaining the build time needs +the container config for each image, and fetching that is subject to strict rate limiting by image +registries (e.g., by [DockerHub][dockerhub-rates]). + +This guide explains how to construct image tags so that the most recent image has the tag that comes +last in alphabetical order. The technique suggested is to put a timestamp or serial number in each +image tag. + +## Formats and alternatives + +The important properties for sorting alphabetically are that the parts of the timestamp go from most +significant to least (e.g., the year down to the second), and that the output is always the same +number of characters. + +Image tags are often shown in user interfaces, so readability matters. Here are some alternatives: + +```bash +$ # seconds-since-epoch (used in the example above) +$ date +%s +1611840548 +$ # date and time (remember ':' is not allowed in a tag) +$ date +%F.%H%M%S +2021-01-28.133158 +``` + +Alternatively, you can use a stable serial number as part of the tag. Some CI platforms will +provide a build number in an environment variable, but that may not be reliable to use as a serial +number -- check the platform documentation. + +A commit count can be a reasonable stand-in for a serial number, if you build an image per commit, +and you don't rewrite the branch in question. + +```bash +$ # commits in branch +$ git --rev-list --count HEAD +1504 +``` + +Beware: this will not give a useful number if you have a shallow clone. + +### Other things to include in the image tag + +It is also handy to quickly trace an image to the branch and commit of its source code. Including +the branch also means you can filter for images from a particular branch. + +In sum, a useful tag format is + + -- + +The branch and tag will usually be made available in a CI platform as environment variables. See + + - [CircleCI's built-in variables `CIRCLE_BRANCH` and `CIRCLE_SHA1`][circle-ci-env] + - [GitHub Actions' `GITHUB_REF` and `GITHUB_SHA`][github-actions-env] + - [Travis CI's `TRAVIS_BRANCH` and `TRAVIS_COMMIT`][travis-env]. + +## Example of a build process with timestamp tagging + +Here is an example of a [GitHub Actions job][gha-syntax] that creates a "build ID" with the git +branch, SHA1, and a timestamp, and uses it as a tag when building an image: + +```yaml +jobs: + build-push: + env: + IMAGE: org/my-app + runs-on: ubuntu-latest + steps: + + - name: Generate build ID + id: prep + run: | + branch=${GITHUB_REF##*/} + sha=${GITHUB_SHA::8} + ts=$(date +%s) + echo "::set-output name=BUILD_ID::${branch}-${sha}-${ts}" + + # These are prerequisites for the docker build step + - name: Set up QEMU + uses: docker/setup-qemu-action@v1 + - name: Set up Docker Buildx + uses: docker/setup-buildx-action@v1 + - name: Login to DockerHub + uses: docker/login-action@v1 + with: + username: ${{ secrets.DOCKERHUB_USERNAME }} + password: ${{ secrets.DOCKERHUB_TOKEN }} + + - name: Build and publish container image with tag + uses: docker/build-push-action@v2 + with: + push: true + context: . + file: ./Dockerfile + tags: | + ${{ env.IMAGE }}:${{ steps.prep.outputs.BUILD_ID }} +``` + +## Using in an `ImagePolicy` object + +When creating an `ImagePolicy` object, you will need to extract just the timestamp part of the tag, +using the `tagFilter` field. You can filter for a particular branch to restrict images to only those +built from that branch. + +Here is an example that filters for only images built from `main` branch, and selects the most +recent according the timestamp (created with `date +%s`): + +``` +apiVersion: image.toolkit.fluxcd.io/v1alpha1 +kind: ImagePolicy +metadata: + name: image-repo-policy + namespace: flux-system +spec: + imageRepositoryRef: + name: image-repo + filterTags: + pattern: '^main-[a-f0-9]+-(?P[0-9]+)' + extract: '$ts' + policy: + alphabetical: + order: asc +``` + +If you don't care about the branch, that part can be a wildcard in the pattern: + +``` +apiVersion: image.toolkit.fluxcd.io/v1alpha1 +kind: ImagePolicy +metadata: + name: image-repo-policy + namespace: flux-system +spec: + imageRepositoryRef: + name: image-repo + filterTags: + pattern: '^.+-[a-f0-9]+-(?P[0-9]+)' + extract: '$ts' + policy: + alphabetical: + order: asc +``` + +[circle-ci-env]: https://circleci.com/docs/2.0/env-vars/#built-in-environment-variables +[github-actions-env]: https://docs.github.com/en/actions/reference/environment-variables#default-environment-variables +[travis-env]: https://docs.travis-ci.com/user/environment-variables/#default-environment-variables +[dockerhub-rates]: https://docs.docker.com/docker-hub/billing/faq/#pull-rate-limiting-faqs +[gha-syntax]: https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions From 43572bba042fd071b73f63d70785ad8aaabc7f89 Mon Sep 17 00:00:00 2001 From: Michael Bridgen Date: Mon, 8 Feb 2021 15:47:31 +0000 Subject: [PATCH 03/13] Rearrange so that observing an auto commit is last Previously, creating an automation object was the last instruction. It is easier to describe what to expect at each step when the last step is to add an update marker in the file to be updated, since the next thing that should happen is that the automation makes an update as a consequence. This commit shifts the sections around so that setting up the GitRepository and ImageUpdateAutomation are done earlier, and migrating each file are done after that, and completes the steps described including checking the expected status at each stage. Signed-off-by: Michael Bridgen --- docs/guides/flux-v1-automation-migration.md | 392 +++++++++++++------- 1 file changed, 266 insertions(+), 126 deletions(-) diff --git a/docs/guides/flux-v1-automation-migration.md b/docs/guides/flux-v1-automation-migration.md index a2e2242a..e80cba04 100644 --- a/docs/guides/flux-v1-automation-migration.md +++ b/docs/guides/flux-v1-automation-migration.md @@ -66,10 +66,16 @@ Flux v2 will ignore them. To migrate to Flux v2 automation, you will need to do three things: - - make sure you are running the automation controllers; and, - - translate Flux v1 annotations to Flux v2 `ImageRepository` and `ImagePolicy` objects, and put - update markers in files; and, - - declare the automation with an `ImageUpdateAutomation` object. + - make sure you are running the automation controllers; then, + - declare the automation with an `ImageUpdateAutomation` object; and, + - migrate each manifest by translate Flux v1 annotations to Flux v2 `ImageRepository` and + `ImagePolicy` objects, and putting update markers in the manifest file. + +### Where to keep `ImageRepository`, `ImagePolicy` and `ImageUpdateAutomation` manifests + +This guide assumes you want to manage automation itself via Flux. In the following sections, +manifests for the objects controlling automation are saved in files, committed to Git, and applied +in the cluster with Flux. A Flux v2 installation will typically have a Git repository structured like this: @@ -81,9 +87,9 @@ A Flux v2 installation will typically have a Git repository structured like this # deployments etc. ``` -The `<...>` is the path to a particular cluster's definitions -- this may be simply `.`, if you did -not supply a `--path` flag when bootstrapping the first time, or something like -`clusters/my-cluster`. To get the files in the right place, set a variable for this path: +The `<...>` is the path to a particular cluster's definitions -- this may be simply `.`, or +something like `clusters/my-cluster`. To get the files in the right place, set a variable for this +path: ```bash $ CLUSTER_PATH=<...> # e.g., "." or "clusters/my-cluster", or ... @@ -93,13 +99,28 @@ $ mkdir ./$AUTO_PATH The file `$CLUSTER_PATH/flux-system/gotk-components.yaml` has definitions of all the Flux v2 controllers and custom resource definitions. The file `gotk-sync.yaml` defines a `GitRepository` and -a `Kustomization` which will sync the Git repository itself. +a `Kustomization` which will sync manifests under `$CLUSTER_PATH/`. To these will be added definitions for automation objects. This guide puts manifest files for automation in `$CLUSTER_PATH/automation/`, but there is no particular structure required by Flux. The automation objects do not have to be in the same namespace as the objects to be updated. +#### Migration on a branch + +This guide assumes you will commit changes to the branch that is synced by Flux, as this is the +simplest way to understand. + +It may be less disruptive to put migration changes on a branch, then merging when you have completed +the migration. You would need to either change the `GitRepository` to point at the migration branch, +or have separate `GitRepository` and `Kustomization` objects for the migrated parts of your Git +repository. The main thing to avoid is syncing the same objects in two different places; e.g., avoid +having Kustomizations that sync both the unmigrated and migrated application configuration. + +%%% TODO decide whether to mention this, explain it in full, or ignore the possibility. + +### Installing the command-line tool `flux` + The command-line tool `flux` will be used below; see [these instructions][install-cli] for how to install it. @@ -170,6 +191,150 @@ $ git push $ flux reconcile kustomization --with-source flux-system ``` +## Controlling automation with an `ImageUpdateAutomation` object + +In Flux v1, automation was run by default. With Flux v2, you have to explicitly tell the controller +which Git repository to update and how to do so. These are defined in an `ImageUpdateAutomation` +object; but first, you need a `GitRepository` with write access, for the automation to use. + +If you followed the [Flux v1 read-only migration guide][flux-v1-migration], you will have a +`GitRepository` defined in the namespace `flux-system`, for syncing to use. This `GitRepository` +will have _read_ access to the Git repository by default, and automation needs _write_ access to +push commits. + +To give it write access, you can replace the secret it refers to. How to do this will depend on what +kind of authentication you used to install Flux v2. + +### Replacing the Git credentials secret + +The secret with Git credentials will be named in the `.spec.secretRef.name` field of the +`GitRepository` object. Say your `GitRepository` is in the _namespace_ `flux-system` and _named_ +`flux-system` (these are the defaults if you used `flux bootstrap`); you can retrieve the secret +name and Git URL with: + +```bash +$ FLUX_NS=flux-system +$ GIT_NAME=flux-system +$ SECRET_NAME=$(kubectl -n $FLUX_NS get gitrepository $GIT_NAME -o jsonpath={.spec.secretRef.name}) +$ GIT_URL=$(kubectl -n $FLUX_NS get gitrepository $GIT_NAME -o jsonpath='{.spec.url}') +$ echo $SECRET_NAME $GIT_URL # make sure they have values +``` + +If you're not sure which kind of credentials you're using, look at the secret: + +```bash +$ kubectl -n $FLUX_NS describe secret $SECRET_NAME +``` + +An entry at `.data.identity` indicates that you are using an SSH key (the [first +section](#replacing-an-ssh-key-secret) below); an entry at `.data.username` indicates you are using +a username and password or token (the [second section](#replacing-a-username-password-secret) +below). + +#### Replacing an SSH key secret + +When using an SSH (deploy) key, create a new key: + +```bash +$ flux create secret git -n $FLUX_NS $SECRET_NAME --url=$GIT_URL +``` + +You will need to copy the public key that's printed out, and install that as a deploy key for your +Git repo **making sure to check the 'All write access' box** (or otherwise give the key write +permissions). Remove the old deploy key. + +#### Replacing a username/password secret + +When you're using a username and password to authenticate, you may be able to change the permissions +associated with that account. + +If not, you will need to create a new access token (e.g., ["Personal Access Token"][github-pat] in +GitHub). In this case, once you have the new token you can replace the secret with the following: + +```bash +$ flux create secret git -n $FLUX_NS $SECRET_NAME \ + --username --password --url $GIT_URL +``` + +#### Checking the new credentials + +To check if your replaced credentials still work, try syncing the `GitRepository` object: + +```bash +$ flux reconcile source git -n $FLUX_NS $GIT_NAME +► annotating GitRepository flux-system in flux-system namespace +✔ GitRepository annotated +◎ waiting for GitRepository reconciliation +✔ GitRepository reconciliation completed +✔ fetched revision main/d537304e8f5f41f1584ca1e807df5b5752b2577e +``` + +When this is successful, it tells you the new credentials have at least read access. + +### Making an automation object + +To set automation running, you create an [`ImageUpdateAutomation`][auto-ref] object. Each object +will update a Git repository, according to the image policies in the namespace. + +Here is an `ImageUpdateAutomation` manifest for the example (note: you will have to supply your own +value for at least the host part of the email address): + +```yaml +$ # the environment variables $AUTO_PATH and $GIT_NAME are set above +$ FLUXBOT_EMAIL=fluxbot@example.com # supply your own host or address here +$ flux create image update my-app-auto \ + --author-name FluxBot --author-email "$FLUXBOT_EMAIL" \ + --git-repo-ref $GIT_NAME --branch main \ + --interval 5m \ + --export > ./$AUTO_PATH/my-app-auto.yaml +$ cat my-app-auto.yaml +--- +apiVersion: image.toolkit.fluxcd.io/v1alpha1 +kind: ImageUpdateAutomation +metadata: + name: my-app-auto + namespace: flux-system +spec: + checkout: + branch: main + gitRepositoryRef: + name: flux-system + commit: + authorEmail: fluxbot@example.com + authorName: FluxBot + interval: 5m0s +``` + +#### Commit and check that the automation object works + +Commit the manifeat file and push: + +```bash +$ git add ./$AUTO_PATH/my-app-auto.yaml +$ git commit -s -m "Add image update automation" +$ git push +# ... +``` + +Then sync and check the object status: + +```bash +$ flux reconcile kustomization --with-source flux-system +► annotating GitRepository flux-system in flux-system namespace +✔ GitRepository annotated +◎ waiting for GitRepository reconciliation +✔ GitRepository reconciliation completed +✔ fetched revision main/401dd3b550f82581c7d12bb79ade389089c6422f +► annotating Kustomization flux-system in flux-system namespace +✔ Kustomization annotated +◎ waiting for Kustomization reconciliation +✔ Kustomization reconciliation completed +✔ reconciled revision main/401dd3b550f82581c7d12bb79ade389089c6422f +$ flux get image update +NAME READY MESSAGE LAST RUN SUSPENDED +my-app-auto True no updates made 2021-02-08T14:53:43Z False +``` + Read on to the next section to see how to change each manifest file to work with Flux v2. ## Migrating each manifest to Flux v2 @@ -223,6 +388,7 @@ the example deployment is `ghcr.io/stefanprodan/podinfo`, which is the image ref tag: ```yaml +$ cat $CLUSTER_PATH/app/my-app.yaml apiVersion: apps/v1 kind: Deployment metadata: @@ -283,7 +449,33 @@ platforms][image-update-tute-clouds]. ##### Committing and checking the ImageRepository -%%% TODO commit, push, reconcile, check the status of the ImageRepository +Add the `ImageRepository` manifest to the Git index and commit it: + +```bash +$ git add ./$AUTO_PATH/podinfo-image.yaml +$ git commit -s -m "Add image repository object for podinfo" +$ git push +# ... +``` + +Now you can sync the new commit, and check that the object is working: + +```bash +$ flux reocncile kustomization --with-source flux-system +► annotating GitRepository flux-system in flux-system namespace +✔ GitRepository annotated +◎ waiting for GitRepository reconciliation +✔ GitRepository reconciliation completed +✔ fetched revision main/fd2fe8a61d4537bcfa349e4d1dbc480ea699ba8a +► annotating Kustomization flux-system in flux-system namespace +✔ Kustomization annotated +◎ waiting for Kustomization reconciliation +✔ Kustomization reconciliation completed +✔ reconciled revision main/fd2fe8a61d4537bcfa349e4d1dbc480ea699ba8a +$ flux get image repository podinfo-image +NAME READY MESSAGE LAST SCAN SUSPENDED +podinfo-image True successful scan, found 16 tags 2021-02-08T14:31:38Z False +``` #### Replacing automation annotations @@ -302,11 +494,11 @@ example, the prefix is `semver:`: These are the prefixes supported in Flux v1, and what to use in Flux v2: -| Flux v1 prefix | Meaning | Flux v2 equivalent | -|------------------|-------------|--------------------| -| `glob:` | Filter for tags matching the glob pattern, then select the newest by build time | [Use timestamped tags](#how-to-use-timestamps-in-image-tags) | -| `regex:` | Filter for tags matching the regular expression, then select the newest by build time |[Use timestamped tags](#how-to-use-timestamp-in-image-tags) | -| `semver:` | Filter for tags that represent versions, and select the highest version in the given range | [Use semver ordering](#how-to-use-semver-image-tags) | +| Flux v1 prefix | Meaning | Flux v2 equivalent | +|----------------|---------|--------------------| +| `glob:` | Filter for tags matching the glob pattern, then select the newest by build time | [Use timestamped tags](#how-to-use-timestamps-in-image-tags) | +| `regex:` | Filter for tags matching the regular expression, then select the newest by build time |[Use timestamped tags](#how-to-use-timestamp-in-image-tags) | +| `semver:` | Filter for tags that represent versions, and select the highest version in the given range | [Use semver ordering](#how-to-use-semver-image-tags) | #### How to use timestamps in image tags @@ -395,13 +587,9 @@ spec: range: '^1.0' ``` -Continue on to the next section to check that your `ImagePolicy` works. - -### Checking the ImagePolicy works +Continue on to the next sections to see an example, and how to check that your `ImagePolicy` works. -%%% TODO commit, push, reconcile, check the status of the ImagePolicy - -#### An ImagePolicy for the example +#### An `ImagePolicy` for the example The example Deployment has annotations using `semver:` as a prefix, so the policy object also uses semver: @@ -427,6 +615,36 @@ spec: range: ^5.0 ``` +#### Checking that the `ImagePolicy` works + +Commit the manifest file, and push: + +```bash +$ git add ./$AUTO_PATH/my-app-policy.yaml +$ git commit -s -m "Add image policy for my-app" +$ git push +# ... +``` + +Then you can reconcile and check that the image policy works: + +```bash +$ flux reconcile kustomization --with-source flux-system +► annotating GitRepository flux-system in flux-system namespace +✔ GitRepository annotated +◎ waiting for GitRepository reconciliation +✔ GitRepository reconciliation completed +✔ fetched revision main/7dcf50222499be8c97e22cd37e26bbcda8f70b95 +► annotating Kustomization flux-system in flux-system namespace +✔ Kustomization annotated +◎ waiting for Kustomization reconciliation +✔ Kustomization reconciliation completed +✔ reconciled revision main/7dcf50222499be8c97e22cd37e26bbcda8f70b95 +$ flux get image policy flux-system +NAME READY MESSAGE LATEST IMAGE +my-app-policy True Latest image tag for 'ghcr.io/stefanprodan/podinfo' resolved to: 5.1.4 ghcr.io/stefanprodan/podinfo:5.1.4 +``` + ### How to mark up files for update The last thing to do in each manifest is to mark the fields that you want to be updated. @@ -491,133 +709,55 @@ field, you can put `:tag` on the end of the name, to replace the value with just image's tag. The [image automation guide][image-update-tute-custom] has examples for `HelmRelease` and other custom resources. -## Controlling automation +### Committing the marker change and checking that automation works -In Flux v1, automation was run by default. With Flux v2, you have to explicitly tell the controller -which Git repository to update and how to do so. These are defined in an `ImageUpdateAutomation` -object; but first, you need a `GitRepository` with write access, for the automation to use. - -If you followed the [Flux v1 read-only migration guide][flux-v1-migration], you will have a -`GitRepository` defined in the namespace `flux-system`, for syncing to use. This `GitRepository` -will have _read_ access to the Git repository by default, and automation needs _write_ access to -push commits. - -To give it write access, you can replace the secret it refers to. How to do this will depend on what -kind of authentication you used to install Flux v2. - -### Replacing the Git credentials secret +Referring to the image policy created earlier, you can see the example Deployment does not use the +most recent image. When you commit the manifest file with the update marker added, you would expect +automation to update the file. -The secret with Git credentials will be named in the `.spec.secretRef.name` field of the -`GitRepository` object. Say your `GitRepository` is in the _namespace_ `flux-system` and _named_ -`flux-system` (these are the defaults if you used `flux bootstrap`); you can retrieve the secret -name and Git URL with: +Commit the change that adds an update marker: ```bash -$ FLUX_NS=flux-system -$ GIT_NAME=flux-system -$ SECRET_NAME=$(kubectl -n $FLUX_NS get gitrepository $GIT_NAME -o jsonpath={.spec.secretRef.name}) -$ GIT_URL=$(kubectl -n $FLUX_NS get gitrepository $GIT_NAME -o jsonpath='{.spec.url}') -$ echo $SECRET_NAME $GIT_URL # make sure they have values +$ git add app/my-app.yaml # the filename of the example +$ git commit -s -m "Add update marker to my-app manifest" +$ git push +# ... ``` -If you're not sure which kind of credentials you're using, look at the secret: +Now to check that the automation makes a change: ```bash -$ kubectl -n $FLUX_NS describe secret $SECRET_NAME +$ flux reconcile image update my-app-auto +► annotating ImageUpdateAutomation my-app-auto in flux-system namespace +✔ ImageUpdateAutomation annotated +◎ waiting for ImageUpdateAutomation reconciliation +✔ ImageUpdateAutomation reconciliation completed +✔ committed and pushed a92a4b654f520c00cb6c46b2d5e4fb4861aa58fc ``` -An entry at `.data.identity` indicates that you are using an SSH key (the [first -section](#replacing-an-ssh-key-secret) below); an entry at `.data.username` indicates you are using -a username and password or token (the [second section](#replacing-a-username-password-secret) -below). +## Troubleshooting -#### Replacing an SSH key secret +If a change was not pushed by the image automation, there's several things you can check: -When using an SSH (deploy) key, create a new key: + - it's possible it made a change that is not reported in the latest status -- pull from the origin + and check the commit log + - check that the name used in the marker corresponds to the namespace and name of an `ImagePolicy` + - check that the `ImageUpdateAutomation` is in the same namespace as the `ImagePolicy` objects + named in markers + - check that the image policy and the image repository are both reported as `Ready` + - check that the credentials referenced by the GitRepository have write permission, and create new + credentials if necessary. -```bash -$ flux create secret git -n $FLUX_NS $SECRET_NAME --url=$GIT_URL -``` - -You will need to copy the public key that's printed out, and install that as a deploy key for your -Git repo **making sure to check the 'All write access' box** (or otherwise give the key write -permissions). Remove the old deploy key. - -#### Replacing a username/password secret - -When you're using a username and password to authenticate, you may be able to change the permissions -associated with that account. - -If not, you will need to create a new access token (e.g., ["Personal Access Token"][github-pat] in -GitHub). In this case, once you have the new token you can replace the secret with the following: - -```bash -$ flux create secret -n $FLUX_NS git $GIT_NAME \ - --username --password --url $GIT_URL -``` - -#### Checking the new credentials - -To check if your replaced credentials still work, try syncing the `GitRepository` object: +As a fallback, you can scan the logs of the automation controller to see if it logged errors: ```bash -$ flux reconcile source git -n $FLUX_NS $GIT_NAME -► annotating GitRepository flux-system in flux-system namespace -✔ GitRepository annotated -◎ waiting for GitRepository reconciliation -✔ GitRepository reconciliation completed -✔ fetched revision main/d537304e8f5f41f1584ca1e807df5b5752b2577e -``` - -Assuming this is successful, it tells you the new credentials have at least read access. - -### Making an automation object - -To set automation running, you create an [`ImageUpdateAutomation`][auto-ref] object. Each object -will update a Git repository according to the image policy objects in the object's namespace. - -Here is an `ImageUpdateAutomation` manifest for the example (note: you will have to supply your own -value for at least the host part of the email address): - -```yaml -$ FLUXBOT_EMAIL=fluxbot@example.com # supply your own host or address here -$ flux create image update my-app-auto \ - --author-name FluxBot --author-email "$FLUXBOT_EMAIL" \ - --git-repo-ref $GIT_NAME --branch main \ - --interval 5m \ - --export > my-app-auto.yaml -$ cat my-app-auto.yaml ---- -apiVersion: image.toolkit.fluxcd.io/v1alpha1 -kind: ImageUpdateAutomation -metadata: - name: my-app-auto - namespace: flux-system -spec: - checkout: - branch: main - gitRepositoryRef: - name: flux-system - commit: - authorEmail: fluxbot@example.com - authorName: FluxBot - interval: 5m0s +$ kubectl logs -n flux-system deploy/image-update-automation ``` -#### Commit and check that the automation object works - -%%% TODO commit, push, reconcile, then check status, then check that it created a commit (if it was going to ..) - # %%% TODO %%% -**Actually create the objects -- sync them from git?** - **Encrypting image pull / certificate secrets** -**What to look at to see if it's working** - -**Where to put repositories and policies (commit them to git!)** - **What to do if you run a build step which loses comments** **How to lock / remove / pause automation** From a541a7ee856db94731f89e7a6cf0f8f57a86c1b4 Mon Sep 17 00:00:00 2001 From: Michael Bridgen Date: Tue, 9 Feb 2021 21:13:30 +0000 Subject: [PATCH 04/13] Remove suggestions of using commit number Using a commit number is trickier than it sounds. It would need to be padded to sort correctly, for one thing. It is better to leave it out than to give an incomplete account. Signed-off-by: Michael Bridgen --- docs/guides/sortable-image-tags.md | 11 ----------- 1 file changed, 11 deletions(-) diff --git a/docs/guides/sortable-image-tags.md b/docs/guides/sortable-image-tags.md index 72bcb2f6..5d5a3554 100644 --- a/docs/guides/sortable-image-tags.md +++ b/docs/guides/sortable-image-tags.md @@ -30,17 +30,6 @@ Alternatively, you can use a stable serial number as part of the tag. Some CI p provide a build number in an environment variable, but that may not be reliable to use as a serial number -- check the platform documentation. -A commit count can be a reasonable stand-in for a serial number, if you build an image per commit, -and you don't rewrite the branch in question. - -```bash -$ # commits in branch -$ git --rev-list --count HEAD -1504 -``` - -Beware: this will not give a useful number if you have a shallow clone. - ### Other things to include in the image tag It is also handy to quickly trace an image to the branch and commit of its source code. Including From 6cd567dc6620694c36457408fbf86702320691e4 Mon Sep 17 00:00:00 2001 From: Michael Bridgen Date: Wed, 10 Feb 2021 13:41:46 +0000 Subject: [PATCH 05/13] Remove draft TODO comments I have moved TODO comments (that still apply) to the PR description. Signed-off-by: Michael Bridgen --- docs/guides/flux-v1-automation-migration.md | 23 +++++---------------- 1 file changed, 5 insertions(+), 18 deletions(-) diff --git a/docs/guides/flux-v1-automation-migration.md b/docs/guides/flux-v1-automation-migration.md index e80cba04..a5837c6b 100644 --- a/docs/guides/flux-v1-automation-migration.md +++ b/docs/guides/flux-v1-automation-migration.md @@ -117,8 +117,6 @@ or have separate `GitRepository` and `Kustomization` objects for the migrated pa repository. The main thing to avoid is syncing the same objects in two different places; e.g., avoid having Kustomizations that sync both the unmigrated and migrated application configuration. -%%% TODO decide whether to mention this, explain it in full, or ignore the possibility. - ### Installing the command-line tool `flux` The command-line tool `flux` will be used below; see [these instructions][install-cli] for how to @@ -745,28 +743,17 @@ If a change was not pushed by the image automation, there's several things you c - check that the `ImageUpdateAutomation` is in the same namespace as the `ImagePolicy` objects named in markers - check that the image policy and the image repository are both reported as `Ready` - - check that the credentials referenced by the GitRepository have write permission, and create new - credentials if necessary. + - check that the credentials referenced by the `GitRepository` object have write permission, and + create new credentials if necessary. As a fallback, you can scan the logs of the automation controller to see if it logged errors: ```bash -$ kubectl logs -n flux-system deploy/image-update-automation +$ kubectl logs -n flux-system deploy/image-automation-controller ``` -# %%% TODO %%% - -**Encrypting image pull / certificate secrets** - -**What to do if you run a build step which loses comments** - -**How to lock / remove / pause automation** - -**Is there an equivalent to the 'ignore' annotation?** - -**Terraform installation?** - -**The Flux v1 migration guide has a branch where you don't put gotk-components.yaml in the repo** +Once you are satisfied that it is working, you can migrate the rest of the manifests using the steps +from ["Migrating each manifest to Flux v2"](#migrating-each-manifest-to-flux-v2) above. [image-update-tute]: https://toolkit.fluxcd.io/guides/image-update/ [imagepolicy-ref]: https://toolkit.fluxcd.io/components/image/imagepolicies/ From daaae0764947496cc36fa3e9514cdc7f76802770 Mon Sep 17 00:00:00 2001 From: Michael Bridgen Date: Wed, 10 Feb 2021 13:52:50 +0000 Subject: [PATCH 06/13] Persuade markdown relative paths are links Signed-off-by: Michael Bridgen --- docs/guides/flux-v1-automation-migration.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/guides/flux-v1-automation-migration.md b/docs/guides/flux-v1-automation-migration.md index a5837c6b..e2e7fe67 100644 --- a/docs/guides/flux-v1-automation-migration.md +++ b/docs/guides/flux-v1-automation-migration.md @@ -759,11 +759,11 @@ from ["Migrating each manifest to Flux v2"](#migrating-each-manifest-to-flux-v2) [imagepolicy-ref]: https://toolkit.fluxcd.io/components/image/imagepolicies/ [helm-auto]: https://docs.fluxcd.io/en/1.21.1/references/helm-operator-integration/#automated-image-detection). [image-update-tute-custom]: https://toolkit.fluxcd.io/guides/image-update/#configure-image-update-for-custom-resources -[flux-v1-migration]: ../flux-v1-migration/ +[flux-v1-migration]: <../flux-v1-migration/> [install-cli]: https://toolkit.fluxcd.io/get-started/#install-the-flux-cli [flux-bootstrap]: https://toolkit.fluxcd.io/guides/installation/#bootstrap [github-pat]: https://docs.github.com/en/github/authenticating-to-github/creating-a-personal-access-token [auto-object-ref]: https://toolkit.fluxcd.io/components/image/imageupdateautomations/ [image-update-tute-creds]: https://toolkit.fluxcd.io/guides/image-update/#configure-image-scanning [image-update-tute-clouds]: https://toolkit.fluxcd.io/guides/image-update/#imagerepository-cloud-providers-authenticatio -[image-tags-guide]: ../sortable-image-tags/ +[image-tags-guide]: <../sortable-image-tags/> From e785971ba85d3e361a89b22f36496b3a19360960 Mon Sep 17 00:00:00 2001 From: Michael Bridgen Date: Fri, 12 Feb 2021 15:04:09 +0000 Subject: [PATCH 07/13] Rewrite to account for numerical sorting Signed-off-by: Michael Bridgen --- docs/guides/flux-v1-automation-migration.md | 12 +++--- docs/guides/sortable-image-tags.md | 48 ++++++++++++++------- 2 files changed, 39 insertions(+), 21 deletions(-) diff --git a/docs/guides/flux-v1-automation-migration.md b/docs/guides/flux-v1-automation-migration.md index e2e7fe67..b448df38 100644 --- a/docs/guides/flux-v1-automation-migration.md +++ b/docs/guides/flux-v1-automation-migration.md @@ -45,7 +45,7 @@ consider. These are dealt with in detail below. Selecting an image by build time is no longer supported. This is the implicit default in Flux v1. In Flux v2, you will need to tag images so that they sort in the order you would like -- [see -below](#how-to-use-timestamps-in-image-tags) for how to do this conveniently. +below](#how-to-use-sortable-image-tags) for how to do this conveniently. #### Fields to update are explicitly marked @@ -494,14 +494,14 @@ These are the prefixes supported in Flux v1, and what to use in Flux v2: | Flux v1 prefix | Meaning | Flux v2 equivalent | |----------------|---------|--------------------| -| `glob:` | Filter for tags matching the glob pattern, then select the newest by build time | [Use timestamped tags](#how-to-use-timestamps-in-image-tags) | -| `regex:` | Filter for tags matching the regular expression, then select the newest by build time |[Use timestamped tags](#how-to-use-timestamp-in-image-tags) | +| `glob:` | Filter for tags matching the glob pattern, then select the newest by build time | [Use sortable tags](#how-to-use-sortable-image-tags) | +| `regex:` | Filter for tags matching the regular expression, then select the newest by build time |[Use sortable tags](#how-to-use-sortable-image-tags) | | `semver:` | Filter for tags that represent versions, and select the highest version in the given range | [Use semver ordering](#how-to-use-semver-image-tags) | -#### How to use timestamps in image tags +#### How to use sortable image tags -To give image tags a useful ordering, you can use a timestamp as part of each image's tag, then sort -alphabetically. +To give image tags a useful ordering, you can use a timestamp or serial number as part of each +image's tag, then sort either alphabetically or numerically. This is a change from Flux v1, in which the build time was fetched from each image's config, and didn't need to be included in the image tag. Therefore, this is likely to require a change to your diff --git a/docs/guides/sortable-image-tags.md b/docs/guides/sortable-image-tags.md index 5d5a3554..5ee56acc 100644 --- a/docs/guides/sortable-image-tags.md +++ b/docs/guides/sortable-image-tags.md @@ -6,36 +6,54 @@ the container config for each image, and fetching that is subject to strict rate registries (e.g., by [DockerHub][dockerhub-rates]). This guide explains how to construct image tags so that the most recent image has the tag that comes -last in alphabetical order. The technique suggested is to put a timestamp or serial number in each -image tag. +last in alphabetical or numerical order. The technique suggested is to put a timestamp or serial +number in each image tag. ## Formats and alternatives -The important properties for sorting alphabetically are that the parts of the timestamp go from most -significant to least (e.g., the year down to the second), and that the output is always the same -number of characters. +The important properties for sorting are that the parts of the timestamp go from most significant to +least (e.g., the year down to the second). For numbers it is best to use numerical order, since this +will work with values of different width (e.g., '12' sorts after '2'). -Image tags are often shown in user interfaces, so readability matters. Here are some alternatives: +Image tags are often shown in user interfaces, so readability matters. Here is an example of a +readable timestamp that will sort well: ```bash -$ # seconds-since-epoch (used in the example above) -$ date +%s -1611840548 $ # date and time (remember ':' is not allowed in a tag) $ date +%F.%H%M%S 2021-01-28.133158 ``` -Alternatively, you can use a stable serial number as part of the tag. Some CI platforms will -provide a build number in an environment variable, but that may not be reliable to use as a serial -number -- check the platform documentation. +You can use a timestamp that sorts as a number, like [Unix +time](https://en.wikipedia.org/wiki/Unix_time): + +``` +$ # seconds since Jan 1 1970 +$ date +%s +1611840548 +``` + +Alternatively, you can use a serial number as part of the tag. Some CI platforms will provide a +build number in an environment variable, but that may not be reliable to use as a serial number -- +check the platform documentation. + +A commit count can be a reasonable stand-in for a serial number, if you build an image per commit +and you don't rewrite the branch in question: + +```bash +$ # commits in branch +$ git --rev-list --count HEAD +1504 +``` + +Beware: this will not give a useful number if you have a shallow clone. ### Other things to include in the image tag It is also handy to quickly trace an image to the branch and commit of its source code. Including the branch also means you can filter for images from a particular branch. -In sum, a useful tag format is +A useful tag format is -- @@ -109,7 +127,7 @@ spec: pattern: '^main-[a-f0-9]+-(?P[0-9]+)' extract: '$ts' policy: - alphabetical: + numerical: order: asc ``` @@ -128,7 +146,7 @@ spec: pattern: '^.+-[a-f0-9]+-(?P[0-9]+)' extract: '$ts' policy: - alphabetical: + numerical: order: asc ``` From 386780ba12341b6c8708c2c3cd3f779727a92f80 Mon Sep 17 00:00:00 2001 From: Michael Bridgen Date: Fri, 12 Feb 2021 15:09:48 +0000 Subject: [PATCH 08/13] Make hrefs absolute and spelt correctly Stray characters here and there threw off the markdown engine. Signed-off-by: Michael Bridgen --- docs/guides/flux-v1-automation-migration.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/guides/flux-v1-automation-migration.md b/docs/guides/flux-v1-automation-migration.md index b448df38..06ba4cbd 100644 --- a/docs/guides/flux-v1-automation-migration.md +++ b/docs/guides/flux-v1-automation-migration.md @@ -757,13 +757,13 @@ from ["Migrating each manifest to Flux v2"](#migrating-each-manifest-to-flux-v2) [image-update-tute]: https://toolkit.fluxcd.io/guides/image-update/ [imagepolicy-ref]: https://toolkit.fluxcd.io/components/image/imagepolicies/ -[helm-auto]: https://docs.fluxcd.io/en/1.21.1/references/helm-operator-integration/#automated-image-detection). +[helm-auto]: https://docs.fluxcd.io/en/1.21.1/references/helm-operator-integration/#automated-image-detection [image-update-tute-custom]: https://toolkit.fluxcd.io/guides/image-update/#configure-image-update-for-custom-resources -[flux-v1-migration]: <../flux-v1-migration/> +[flux-v1-migration]: https://toolkit.fluxcd.io/guides/flux-v1-migration/ [install-cli]: https://toolkit.fluxcd.io/get-started/#install-the-flux-cli [flux-bootstrap]: https://toolkit.fluxcd.io/guides/installation/#bootstrap [github-pat]: https://docs.github.com/en/github/authenticating-to-github/creating-a-personal-access-token [auto-object-ref]: https://toolkit.fluxcd.io/components/image/imageupdateautomations/ [image-update-tute-creds]: https://toolkit.fluxcd.io/guides/image-update/#configure-image-scanning -[image-update-tute-clouds]: https://toolkit.fluxcd.io/guides/image-update/#imagerepository-cloud-providers-authenticatio -[image-tags-guide]: <../sortable-image-tags/> +[image-update-tute-clouds]: https://toolkit.fluxcd.io/guides/image-update/#imagerepository-cloud-providers-authentication +[image-tags-guide]: https://toolkit.fluxcd.io/guides/sortable-image-tags/ From 5e1f6f7317533b37a13ff29c6cecebdf4df2acc4 Mon Sep 17 00:00:00 2001 From: Michael Bridgen Date: Fri, 12 Feb 2021 15:29:02 +0000 Subject: [PATCH 09/13] Fix up internal links Signed-off-by: Michael Bridgen --- docs/guides/flux-v1-automation-migration.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/guides/flux-v1-automation-migration.md b/docs/guides/flux-v1-automation-migration.md index 06ba4cbd..f75af74b 100644 --- a/docs/guides/flux-v1-automation-migration.md +++ b/docs/guides/flux-v1-automation-migration.md @@ -133,7 +133,7 @@ guide][flux-v1-migration]. - If you used `flux install` to install the controllers directly, use [After migrating Flux v1 in place](#after-migrating-flux-v1-in-place); - If you used `flux install` and exported the configuration to a file, use [After committing Flux - v2 configuration to Git](#after-committing-flux-v2-configuration-to-git). + v2 configuration to Git](#after-committing-a-flux-v2-configuration-to-git). ### After `flux bootstrap` @@ -226,7 +226,7 @@ $ kubectl -n $FLUX_NS describe secret $SECRET_NAME An entry at `.data.identity` indicates that you are using an SSH key (the [first section](#replacing-an-ssh-key-secret) below); an entry at `.data.username` indicates you are using -a username and password or token (the [second section](#replacing-a-username-password-secret) +a username and password or token (the [second section](#replacing-a-usernamepassword-secret) below). #### Replacing an SSH key secret @@ -548,7 +548,7 @@ timestamp part of the tag will be extracted and sorted alphabetically in ascendi reference docs][imagepolicy-ref] for more examples. Once you have made sure you have image tags and an `ImagePolicy` that works, jump ahead to [Checking -the ImagePolicy works](#checking-the-image-policy-works). +the ImagePolicy works](#checking-that-the-image-policy-works). ### How to use SemVer image tags From 6e1672f73c8bbda764a2ca66cb9ea0ef78842083 Mon Sep 17 00:00:00 2001 From: Hidde Beydals Date: Fri, 12 Feb 2021 16:47:16 +0100 Subject: [PATCH 10/13] Change policy example to numerical in asc order Signed-off-by: Hidde Beydals --- docs/guides/flux-v1-automation-migration.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/guides/flux-v1-automation-migration.md b/docs/guides/flux-v1-automation-migration.md index f75af74b..b9e53da3 100644 --- a/docs/guides/flux-v1-automation-migration.md +++ b/docs/guides/flux-v1-automation-migration.md @@ -537,14 +537,14 @@ spec: pattern: '^main-[a-f0-9]+-(?P[0-9]+)' extract: '$ts' policy: - alphabetical: + numerical: order: asc ``` The `.spec.pattern` field gives a regular expression that a tag must match to be included. The `.spec.extract` field gives a replacement pattern that can refer back to capture groups in the filter pattern. The extracted values are sorted to find the selected image tag. In this case, the -timestamp part of the tag will be extracted and sorted alphabetically in ascending order. See [the +timestamp part of the tag will be extracted and sorted numerically in ascending order. See [the reference docs][imagepolicy-ref] for more examples. Once you have made sure you have image tags and an `ImagePolicy` that works, jump ahead to [Checking From 8edc4bd24b5d32bb3c039fdc26020f31d1cb1b7e Mon Sep 17 00:00:00 2001 From: Hidde Beydals Date: Fri, 12 Feb 2021 16:49:33 +0100 Subject: [PATCH 11/13] Add missing link to SemVer spec Signed-off-by: Hidde Beydals --- docs/guides/flux-v1-automation-migration.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/guides/flux-v1-automation-migration.md b/docs/guides/flux-v1-automation-migration.md index b9e53da3..eed973e8 100644 --- a/docs/guides/flux-v1-automation-migration.md +++ b/docs/guides/flux-v1-automation-migration.md @@ -767,3 +767,4 @@ from ["Migrating each manifest to Flux v2"](#migrating-each-manifest-to-flux-v2) [image-update-tute-creds]: https://toolkit.fluxcd.io/guides/image-update/#configure-image-scanning [image-update-tute-clouds]: https://toolkit.fluxcd.io/guides/image-update/#imagerepository-cloud-providers-authentication [image-tags-guide]: https://toolkit.fluxcd.io/guides/sortable-image-tags/ +[semver]: https://semver.org From e034ec3207cca70093aa23609487114202a191ec Mon Sep 17 00:00:00 2001 From: Hidde Beydals Date: Fri, 12 Feb 2021 16:53:32 +0100 Subject: [PATCH 12/13] Add missing link to image update automation ref Signed-off-by: Hidde Beydals --- docs/guides/flux-v1-automation-migration.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/guides/flux-v1-automation-migration.md b/docs/guides/flux-v1-automation-migration.md index eed973e8..6f21ce25 100644 --- a/docs/guides/flux-v1-automation-migration.md +++ b/docs/guides/flux-v1-automation-migration.md @@ -767,4 +767,5 @@ from ["Migrating each manifest to Flux v2"](#migrating-each-manifest-to-flux-v2) [image-update-tute-creds]: https://toolkit.fluxcd.io/guides/image-update/#configure-image-scanning [image-update-tute-clouds]: https://toolkit.fluxcd.io/guides/image-update/#imagerepository-cloud-providers-authentication [image-tags-guide]: https://toolkit.fluxcd.io/guides/sortable-image-tags/ +[auto-ref]: https://toolkit.fluxcd.io/components/image/imageupdateautomations/ [semver]: https://semver.org From 30c1c5c3d3b3227e6ffe1fea3dcb3068f2781a3c Mon Sep 17 00:00:00 2001 From: Hidde Beydals Date: Fri, 12 Feb 2021 16:53:56 +0100 Subject: [PATCH 13/13] Link to image automation guides in menu Signed-off-by: Hidde Beydals --- mkdocs.yml | 2 ++ 1 file changed, 2 insertions(+) diff --git a/mkdocs.yml b/mkdocs.yml index 900c0f45..ca41115b 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -48,6 +48,7 @@ nav: - Get Started: get-started/index.md - Migration: - Migrate from Flux v1: guides/flux-v1-migration.md + - Migrate from Flux v1 image update automation: guides/flux-v1-automation-migration.md - Migrate from the Helm Operator: guides/helm-operator-migration.md - Guides: - Installation: guides/installation.md @@ -58,6 +59,7 @@ nav: - Sealed Secrets: guides/sealed-secrets.md - Mozilla SOPS: guides/mozilla-sops.md - Automate image updates to Git: guides/image-update.md + - Sortable image tags to use with automation: guides/sortable-image-tags.md - Toolkit Components: - Overview: components/index.md - Source Controller: