aboutsummaryrefslogtreecommitdiff
path: root/tests/docker/docker.py
AgeCommit message (Collapse)AuthorFilesLines
2023-10-17tests/docker: avoid invalid escape in Python stringPaolo Bonzini1-1/+1
This is an error in Python 3.12; fix it by using a raw string literal. Cc: qemu-stable@nongnu.org Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-03-13tests/docker: fix a win32 error due to portabilityMarc-André Lureau1-3/+3
docker.py is run during configure, and produces an error: No module named 'pwd'. Use a more portable and recommended alternative to lookup the user "login name". Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Message-Id: <20230306122751.2355515-4-marcandre.lureau@redhat.com>
2022-09-20tests/docker: remove FROM qemu/ support from docker.pyAlex Bennée1-28/+10
We want to migrate from docker.py to building our images directly with docker/podman. Before we get there we need to make sure we don't re-introduce our layered builds so bug out if we see FROM qemu/ in a Dockerfile. Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Acked-by: Thomas Huth <thuth@redhat.com> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Message-Id: <20220914155950.804707-30-alex.bennee@linaro.org>
2022-04-20tests/docker: remove SKIP_DOCKER_BUILDPaolo Bonzini1-57/+0
It is now unused. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Message-Id: <20220401141326.1244422-17-pbonzini@redhat.com> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Message-Id: <20220419091020.3008144-20-alex.bennee@linaro.org>
2021-07-14tests/docker: don't use BUILDKIT in GitLab eitherDaniel P. Berrangé1-1/+3
Using BUILDKIT breaks with certain container registries such as CentOS, with docker build reporting an error such as failed to solve with frontend dockerfile.v0: failed to build LLB: failed to load cache key: unexpected status code https://registry.centos.org/v2/centos/manifests/7: 403 Forbidden Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Willian Rampazzo <willianr@redhat.com> Message-Id: <20210623142245.307776-3-berrange@redhat.com> Message-Id: <20210709143005.1554-11-alex.bennee@linaro.org>
2021-05-18tests/docker: add "fetch" sub-commandAlex Bennée1-0/+17
This simply wraps up fetching a build from the registry and tagging it as the local build. Reviewed-by: Willian Rampazzo <willianr@redhat.com> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Message-Id: <20210512102051.12134-6-alex.bennee@linaro.org>
2021-05-18tests/docker: allow "update" to add the current userAlex Bennée1-1/+11
The current user functionality is used for cross compiling to avoid complications with permissions when building test programs. However for images that come from the registry we still need the ability to add the user after the fact. Reviewed-by: Willian Rampazzo <willianr@redhat.com> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Message-Id: <20210512102051.12134-5-alex.bennee@linaro.org>
2021-05-18tests/docker: make executable an optional argument to "update"Alex Bennée1-27/+29
We're going to extend the abilities of the command shortly. Reviewed-by: Willian Rampazzo <willianr@redhat.com> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Message-Id: <20210512102051.12134-4-alex.bennee@linaro.org>
2021-05-18tests/docker: fix copying of executable in "update"Alex Bennée1-1/+8
We have the same symlink chasing problem when doing an "update" operation. Fix that. Based-on: 5e33f7fead ("tests/docker: better handle symlinked libs") Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Willian Rampazzo <willianr@redhat.com> Message-Id: <20210512102051.12134-3-alex.bennee@linaro.org>
2021-02-08tests/docker: add a docker-exec-copy-testAlex Bennée1-1/+6
This provides test machinery for checking the QEMU copying logic works properly. It takes considerably less time to run than starting a debootstrap only for it to fail later. I considered adding a remove command to docker.py but figured that might be gold plating given the relative size of the containers compared to the ones with actual stuff in them. Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Message-Id: <20210202134001.25738-7-alex.bennee@linaro.org>
2021-02-08tests/docker: preserve original name when copying libsAlex Bennée1-3/+4
While it is important we chase down the symlinks to copy the correct data we can confuse the kernel by renaming the interpreter to what is in the binary. Extend _copy_with_mkdir to preserve the original name of the file when asked. Fixes: 5e33f7fead ("tests/docker: better handle symlinked libs") Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Message-Id: <20210202134001.25738-5-alex.bennee@linaro.org>
2021-02-08tests/docker: make _copy_with_mkdir accept missing filesAlex Bennée1-1/+6
Depending on the linker/ldd setup we might get a file with no path. Typically this is the psuedo library linux-vdso.so which doesn't actually exist on the disk. Rather than try and catch these distro specific edge cases just shout about it and try and continue. Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Message-Id: <20210202134001.25738-4-alex.bennee@linaro.org>
2021-02-08tests/docker: Fix _get_so_libs() for docker-binfmt-imagePhilippe Mathieu-Daudé1-1/+1
Fix a variable rename mistake from commit 5e33f7fead5: Traceback (most recent call last): File "./tests/docker/docker.py", line 710, in <module> sys.exit(main()) File "./tests/docker/docker.py", line 706, in main return args.cmdobj.run(args, argv) File "./tests/docker/docker.py", line 489, in run _copy_binary_with_libs(args.include_executable, File "./tests/docker/docker.py", line 149, in _copy_binary_with_libs libs = _get_so_libs(src) File "./tests/docker/docker.py", line 123, in _get_so_libs libs.append(s.group(1)) NameError: name 's' is not defined Fixes: 5e33f7fead5 ("tests/docker: better handle symlinked libs") Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Message-Id: <20210119050149.516910-1-f4bug@amsat.org> Message-Id: <20210202134001.25738-2-alex.bennee@linaro.org>
2021-01-18Add newline when generating DockerfileAlessandro Di Federico1-2/+2
Signed-off-by: Alessandro Di Federico <ale@rev.ng> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Willian Rampazzo <willianr@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> Message-Id: <1610080146-14968-36-git-send-email-tsimpson@quicinc.com> Message-Id: <20210114165730.31607-5-alex.bennee@linaro.org>
2020-09-18docker.py: always use --rmPaolo Bonzini1-2/+2
Avoid that containers pile up. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Acked-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
2020-07-27tests/docker: fix update command due to python3 str/bytes distinctionAlex Bennée1-6/+7
Does this seem convoluted to you? It feels a little complicated to me. Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Message-Id: <20200724064509.331-10-alex.bennee@linaro.org>
2020-07-15docker.py: fix fetching of FROM layersAlex Bennée1-5/+11
This worked on a system that was already bootstrapped because the stage 2 images already existed even if they wouldn't be used. What we should have pulled down was the FROM line containers first because building on gitlab doesn't have the advantage of using our build system to build the pre-requisite bits. We still pull the image we want to build just in case we can use the cached data. Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Message-Id: <20200713200415.26214-4-alex.bennee@linaro.org>
2020-07-11tests/docker: fall back more gracefully when pull failsAlex Bennée1-5/+6
I only spotted this in the small window between my testing with my registry while waiting for the gitlab PR to go in. As we pre-pull the registry image we know if that fails there isn't any point attempting to use the cache. Fall back to the way we used to do it at that point. Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Message-Id: <20200709141327.14631-12-alex.bennee@linaro.org>
2020-07-11tests/docker: add --registry support to toolingAlex Bennée1-6/+38
This allows us to point the tools towards a registry from which they can grab pre-built layers instead of doing everything from scratch each time. To enable this we need to be using the DOCKER_BUILDKIT engine. Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Message-Id: <20200701135652.1366-25-alex.bennee@linaro.org>
2020-07-11tests/docker: change tag naming scheme of our imagesAlex Bennée1-1/+1
We've been misusing the tag naming scheme for some time by overloading the post : section with the image type. Really it should be saved for the revision of that particular build. Move the details to the other side so we have: qemu/image-name with the implied :latest version added by the tooling. Suggested-by: Daniel P. Berrangé <berrange@redhat.com> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> Message-Id: <20200701135652.1366-18-alex.bennee@linaro.org>
2020-06-10docker.py/build: support binary files in --extra-filesPaolo Bonzini1-2/+6
Read the --extra-files in binary mode to avoid encoding errors. Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2020-06-10docker.py/build: support -t and -f argumentsPaolo Bonzini1-3/+3
The docker.py command line is subtly different from docker and podman's, in that the tag and Dockerfile are passed via positional arguments. Remove this gratuitous difference and just parse -f and -t. -f was previously used by --extra-files, only keep the long option. Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2020-05-31python: remove more instances of sys.version_infoJohn Snow1-2/+3
We guarantee 3.5+ everywhere; remove more dead checks. In general, try to avoid using version checks and instead prefer to attempt behavior when possible. Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> Message-Id: <20200514035230.25756-1-jsnow@redhat.com> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
2020-02-04tests/docker: better handle symlinked libsAlex Bennée1-7/+9
When we are copying we want to ensure we grab the first resolution (the found in path section). However even that binary might be a symlink so lets make sure we chase the symlinks to copy the right binary to where it can be found. Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com> Reviewed-by: Robert Foley <robert.foley@linaro.org> Message-Id: <20200203090932.19147-3-alex.bennee@linaro.org>
2019-09-26podman: fix command invocationJohn Snow1-1/+1
Oops; there's no argv here. Signed-off-by: John Snow <jsnow@redhat.com> Message-Id: <20190913193821.17756-1-jsnow@redhat.com> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Cleber Rosa <crosa@redhat.com> Tested-by: Cleber Rosa <crosa@redhat.com> Reviewed-by: Thomas Huth <thuth@redhat.com> Tested-by: Thomas Huth <thuth@redhat.com>
2019-09-26tests/docker: reduce scary warnings by cleaning up clean upAlex Bennée1-18/+16
There was in the clean-up code caused by attempting to inspect images which finished before we got there. Clean up the clean up code by: - only track the one instance at a time - use --filter for docker ps instead of doing it by hand - just call docker rm -f to be done with it - use uuid.uuid4() for a random uid Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
2019-09-10tests/docker: fix final missing .encode when parsing solibsAlex Bennée1-1/+1
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
2019-09-10tests/docker: handle missing encoding keyword for subprocess.check_outputAlex Bennée1-4/+10
This was only added in Python 3.6 and not all the build hosts have that recent a python3. However we still need to ensure everything is returns as a unicode string so checks higher up the call chain don't barf. Signed-off-by: Alex Bennée <alex.bennee@linaro.org> fixup! tests/docker: handle missing encoding keyword for subprocess.check_output
2019-09-10tests/docker: fix "cc" command to work with podmanAlex Bennée1-13/+13
Podman requires a little bit of additional magic to the uid mapping which was already done for the normal RunCommand. We simplify the logic by pushing it directly into the Docker::run method to avoid instantiating an extra Docker() object and ensure the CC command always runs as the current user. Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
2019-09-10tests/docker: Use --userns=keep-id for podmanJohn Snow1-3/+1
The workaround that attempts to accomplish the same result as --userns=keep-id does not appear to work well with UIDs much above 1000 (like mine, which is above 20000.) Since we have official support for this "trick" now, use the supported method. Signed-off-by: John Snow <jsnow@redhat.com> Message-Id: <20190904232451.26466-1-jsnow@redhat.com> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
2019-09-10configure: clean-up container cross compile detectAlex Bennée1-3/+4
The introduction of podman support inadvertently broke configure's detect of the container support as the configure probe didn't specify an engine type. To fix this in docker.py: - only (re)set USE_ENGINE if --engine is specified - enhance the output so docker is no longer just yes In the configure script we can at least start cleaning up the detecting and naming of variables. To avoid too much churn the conversion of the various make DOCKER_foo variables has been left for future clean-ups. Fixes: 9459f754134b Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Tested-by: David Hildenbrand <david@redhat.com>
2019-09-02tests/docker: upgrade docker.py to python3Alex Bennée1-13/+14
The recent podman changes (9459f754134bb) imported enum which is part of the python3 standard library but only available as an external library for python2. This causes problems on the fairly restricted environment such as shippable. Lets bite the bullet and make the script a fully python3 one. To that end: - drop the from __future__ import (we are there now ;-) - avoid the StringIO import hack - be consistent with the mode we read/write dockerfiles - s/iteritems/items/ - ensure check_output returns strings for processing Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> Cc: Marc-André Lureau <marcandre.lureau@redhat.com>
2019-08-22docker.py: add podman supportMarc-André Lureau1-5/+43
Add a --engine option to select either docker, podman or auto. Among other advantages, podman allows to run rootless & daemonless containers, fortunately sharing compatible CLI with docker. With current podman, we have to use a uidmap trick in order to be able to rw-share the ccache directory with the container user. With a user 1000, the default mapping is: 1000 (host) -> 0 (container). So write access to /var/tmp/ccache ends will end with permission denied error. With "--uidmap 1000:0:1 --uidmap 0:1:1000", the mapping is: 1000 (host) -> 0 (container, 1st namespace) -> 1000 (container, 2nd namespace). (the rest is mumbo jumbo to avoid holes in the range of UIDs) A future podman version may have an option such as --userns-keep-uid. Thanks to Debarshi Ray <rishi@redhat.com> for the help! Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com> Acked-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2019-08-22docker.py: add --run-as-current-userMarc-André Lureau1-0/+5
(podman will need further tweaks) Suggested-by: Paolo Bonzini <pbonzini@redhat.com> Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> Reviewed-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2019-02-08tests: PEP8 cleanup of docker.py, mostly white spaceAlex Bennée1-13/+35
My editor keeps putting squiggly lines under a bunch of the python lines to remind me how non-PEP8 compliant it is. Clean that up so it's easier to spot new errors. Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
2019-02-08tests: docker.py be even smarter with persistent binfmt_miscAlex Bennée1-18/+38
If we have a persistent mapping we don't need the QEMU binary copied into the container as the kernel has already opened the file and will pass the fd in. However the support libraries will still need to be there. Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
2019-02-08tests: make docker.py check for persistent configsAlex Bennée1-0/+5
binfmt_misc configured with the "F" flag opens the interpreter at config time. This means it can use an already open file-descriptor to run QEMU so there is no point trying to copy the binary into a container. Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
2019-02-08tests: make docker.py update use configured binfmt pathAlex Bennée1-10/+15
When copying a QEMU binary into a linux-user docker image we should check what the current configured binfmt_misc path is rather than just assuming "/usr/bin/qemu-bin". Obviously if the user changes the configuration afterwards they will break their images again. Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
2018-10-26tests: Fix typos in comments and help message (found by codespell)Stefan Weil1-2/+2
Fix also a grammar issue. Signed-off-by: Stefan Weil <sw@weilnetz.de> Reviewed-by: Alex Bennée <alex.bennee@linaro.org> Message-Id: <20180713054755.23323-1-sw@weilnetz.de> Signed-off-by: Laurent Vivier <laurent@vivier.eu>
2018-10-03Revert "docker.py: Python 2.6 argparse compatibility"Eduardo Habkost1-3/+1
This reverts commit c2d3189667409561772e8c1e5615c5166cd8aa2c. scripts/argparse.py was removed from the tree, so we don't need this hack anymore. Signed-off-by: Eduardo Habkost <ehabkost@redhat.com> Message-Id: <20180618225131.13113-3-ehabkost@redhat.com> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com> Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
2018-07-24docker: perform basic binfmt_misc validation in docker.pyAlex Bennée1-0/+30
Setting up binfmt_misc is outside of the scope of the docker.py script but we can at least validate it with any given executable so we have a more useful error message than the sed line of deboostrap failing cryptically. Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Reported-by: Richard Henderson <richard.henderson@linaro.org>
2018-07-24docker: fail more gracefully on docker.py checkAlex Bennée1-1/+6
As this is called directly from the Makefile while determining dependencies and it is possible the user was configured in one window but not have credentials in the other. Let's catch the Exceptions and deal with it quietly. Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Reported-by: Peter Maydell <peter.maydell@linaro.org> Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
2018-07-05docker: add special handling for FROM:debian-%-user targetsAlex Bennée1-0/+4
These will have been build with debootstrap so we need to check against the debian-bootstrap dockerfile. This does mean sticking to debian-FOO-user as the naming conventions for boot-strapped images. The actual cross image is built on top. Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
2018-06-20docker: docker.py adding age check commandAlex Bennée1-10/+36
This is useful for querying if an image is too old. Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
2018-06-20docker: docker.py add check sub-commandAlex Bennée1-1/+37
This command allows you to check if we need to re-build a docker image. If the image isn't in the repository or the checksums don't match then we return false and some text (for processing in makefiles). Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
2018-06-20docker: docker.py don't conflate checksums for extra_filesAlex Bennée1-3/+5
This just gets confusing especially as the helper function doesn't even take into account any extra files (or the executable). Currently the actual check just ignores them and also passes the result through _dockerfile_preprocess so we fix that too. Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
2018-06-20docker: docker.py use "version" to probe usageAlex Bennée1-1/+3
The "images" command is a fairly heavyweight command to run as it involves searching the whole docker file-system inventory. On a machine with a lot of images this makes start-up fairly expensive. Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
2018-06-20docker: allow "cc" command to run in user contextAlex Bennée1-0/+4
Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Fam Zheng <famz@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
2018-06-20docker: extend "cc" command to accept compilerAlex Bennée1-1/+3
When calling our cross-compilation images we want to call something other than the default cc. Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Reviewed-by: Fam Zheng <famz@redhat.com>
2018-06-20docker: Add "cc" subcommandAlex Bennée1-0/+25
Signed-off-by: Fam Zheng <famz@redhat.com> [AJB: add if args.paths check] Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
generated by cgit v1.1 at 2025-04-24 23:20:22 +0000