lcitool/manifest: eliminate redundant container rebuilds

Currently we rebuild containers on every pipeline that is run.

We only ever keep one tag 'latest' per container in the
registry, regardless of branch. This is done to avoid filling
the registry for per-branch tags. The cost is that we must
rebuild the container on every branch push, since the current
'latest' tag might reflect an unrelated branch. The use of a
single tag also creates a race if pipelines from different
branches get triggered concurrently. The cost is somewhat
mitigated by using layer caching during build.

Historically this was handled the same way for forks and
upstream, but for upstream it is overkill. We don't make use
of CI against anything except the default branch, so the
content of the 'latest' tag always reflects current HEAD.

Given this there are only three cases where the upstream needs
to re-build containers

  - The Dockerfile / container-template.yml files changed
  - To pick up new packages published by the distro
  - A project member deleted a container from the registry

IOW, in upstream:

  - Most push events don't need to rebuild the 'latest' tag,
    only those with relevant file paths changed.

  - Only certain scheduled pipelines need to trigger a
    container rebuild. One a week is fine.

  - Manually triggered pipelines may need to rebuild

This patch set rules against the .container_job template to cut
down on jobs in upstream. To force the container builds in a
pipeline, the LIBVIRT_CI_CONTAINERS=1 variable should be set.

When updating across this change, project admins will need to
setup a scheduled job todo the weekly container rebuild.

Since the container build jobs will not exist in certain scenarios,
we now need to make the deps in the build jobs optional.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>

3 files changed, 70 insertions, 9 deletions

diff --git a/guests/lcitool/lcitool/gitlab.py b/guests/lcitool/lcitool/gitlab.py
index c1e098d..7a58c92 100644
--- a/guests/lcitool/lcitool/gitlab.py
+++ b/guests/lcitool/lcitool/gitlab.py
@@ -15,6 +15,18 @@ def includes(paths):
 def container_template(namespace, project, cidir):
     return textwrap.dedent(
         f"""
+        # For upstream
+        #
+        #   - Push to default branch:
+        #       -> rebuild if dockerfile changed, no cache
+        #   - Otherwise
+        #       -> rebuild if LIBVIRT_CI_CONTAINERS=1, no cache,
+        #          to pick up new published distro packages or
+        #          recover from deleted tag
+        #
+        # For forks
+        #   - Always rebuild, with cache
+        #
         .container_job:
           image: docker:stable
           stage: containers
@@ -27,11 +39,27 @@ def container_template(namespace, project, cidir):
             - docker info
             - docker login "$CI_REGISTRY" -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD"
           script:
-            - docker pull "$TAG" || docker pull "$COMMON_TAG" || true
-            - docker build --cache-from "$TAG" --cache-from "$COMMON_TAG" --tag "$TAG" -f "{cidir}/containers/$NAME.Dockerfile" {cidir}/containers
+            - if test $CI_PROJECT_NAMESPACE = "{namespace}";
+              then
+                docker build --tag "$TAG" -f "{cidir}/containers/$NAME.Dockerfile" {cidir}/containers ;
+              else
+                docker pull "$TAG" || docker pull "$COMMON_TAG" || true ;
+                docker build --cache-from "$TAG" --cache-from "$COMMON_TAG" --tag "$TAG" -f "{cidir}/containers/$NAME.Dockerfile" {cidir}/containers ;
+              fi
             - docker push "$TAG"
           after_script:
             - docker logout
+          rules:
+            - if: '$CI_PROJECT_NAMESPACE == "{namespace}" && $CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH'
+              when: on_success
+              changes:
+               - {cidir}/gitlab/container-templates.yml
+               - {cidir}/containers/$NAME.Dockerfile
+            - if: '$CI_PROJECT_NAMESPACE == "{namespace}" && $LIBVIRT_CI_CONTAINERS == "1"'
+              when: on_success
+            - if: '$CI_PROJECT_NAMESPACE == "{namespace}"'
+              when: never
+            - when: on_success
         """)
 
 
@@ -224,7 +252,8 @@ def _build_job(target, arch, suffix, variables, template, allow_failure, artifac
         {arch}-{target}{suffix}:
           extends: {template}
           needs:
-            - {arch}-{target}-container
+            - job: {arch}-{target}-container
+              optional: true
           allow_failure: {allow_failure}
         """) + format_variables(variables) + format_artifacts(artifacts)
 
diff --git a/guests/lcitool/tests/data/manifest/out/ci/gitlab/builds.yml b/guests/lcitool/tests/data/manifest/out/ci/gitlab/builds.yml
index 3e1f87b..af6f84a 100644
--- a/guests/lcitool/tests/data/manifest/out/ci/gitlab/builds.yml
+++ b/guests/lcitool/tests/data/manifest/out/ci/gitlab/builds.yml
@@ -3,7 +3,8 @@
 x86_64-centos-stream-9:
   extends: .native_build_job
   needs:
-    - x86_64-centos-stream-9-container
+    - job: x86_64-centos-stream-9-container
+      optional: true
   allow_failure: false
   variables:
     NAME: centos-stream-9
@@ -12,7 +13,8 @@ x86_64-centos-stream-9:
 x86_64-fedora-rawhide-clang:
   extends: .native_build_job
   needs:
-    - x86_64-fedora-rawhide-container
+    - job: x86_64-fedora-rawhide-container
+      optional: true
   allow_failure: false
   variables:
     CC: clang
@@ -25,7 +27,8 @@ x86_64-fedora-rawhide-clang:
 i686-debian-sid:
   extends: .cross_build_job
   needs:
-    - i686-debian-sid-container
+    - job: i686-debian-sid-container
+      optional: true
   allow_failure: false
   variables:
     CROSS: i686
@@ -40,7 +43,8 @@ i686-debian-sid:
 mingw32-fedora-rawhide:
   extends: .cross_build_job
   needs:
-    - mingw32-fedora-rawhide-container
+    - job: mingw32-fedora-rawhide-container
+      optional: true
   allow_failure: false
   variables:
     CROSS: mingw32
diff --git a/guests/lcitool/tests/data/manifest/out/ci/gitlab/container-templates.yml b/guests/lcitool/tests/data/manifest/out/ci/gitlab/container-templates.yml
index 47bed1e..cc2f8ac 100644
--- a/guests/lcitool/tests/data/manifest/out/ci/gitlab/container-templates.yml
+++ b/guests/lcitool/tests/data/manifest/out/ci/gitlab/container-templates.yml
@@ -1,3 +1,15 @@
+# For upstream
+#
+#   - Push to default branch:
+#       -> rebuild if dockerfile changed, no cache
+#   - Otherwise
+#       -> rebuild if LIBVIRT_CI_CONTAINERS=1, no cache,
+#          to pick up new published distro packages or
+#          recover from deleted tag
+#
+# For forks
+#   - Always rebuild, with cache
+#
 .container_job:
   image: docker:stable
   stage: containers
@@ -10,8 +22,24 @@
     - docker info
     - docker login "$CI_REGISTRY" -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD"
   script:
-    - docker pull "$TAG" || docker pull "$COMMON_TAG" || true
-    - docker build --cache-from "$TAG" --cache-from "$COMMON_TAG" --tag "$TAG" -f "ci/containers/$NAME.Dockerfile" ci/containers
+    - if test $CI_PROJECT_NAMESPACE = "test-group";
+      then
+        docker build --tag "$TAG" -f "ci/containers/$NAME.Dockerfile" ci/containers ;
+      else
+        docker pull "$TAG" || docker pull "$COMMON_TAG" || true ;
+        docker build --cache-from "$TAG" --cache-from "$COMMON_TAG" --tag "$TAG" -f "ci/containers/$NAME.Dockerfile" ci/containers ;
+      fi
     - docker push "$TAG"
   after_script:
     - docker logout
+  rules:
+    - if: '$CI_PROJECT_NAMESPACE == "test-group" && $CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH'
+      when: on_success
+      changes:
+       - ci/gitlab/container-templates.yml
+       - ci/containers/$NAME.Dockerfile
+    - if: '$CI_PROJECT_NAMESPACE == "test-group" && $LIBVIRT_CI_CONTAINERS == "1"'
+      when: on_success
+    - if: '$CI_PROJECT_NAMESPACE == "test-group"'
+      when: never
+    - when: on_success