From 0da9856851dcca09222a1467e16ddd05dc66e460 Mon Sep 17 00:00:00 2001 From: Nir Soffer Date: Fri, 19 Feb 2021 18:07:52 +0200 Subject: nbd: server: Report holes for raw images When querying image extents for raw image, qemu-nbd reports holes as zero: $ qemu-nbd -t -r -f raw empty-6g.raw $ qemu-img map --output json nbd://localhost [{ "start": 0, "length": 6442450944, "depth": 0, "zero": true, "data": true, "offset": 0}] $ qemu-img map --output json empty-6g.raw [{ "start": 0, "length": 6442450944, "depth": 0, "zero": true, "data": false, "offset": 0}] Turns out that qemu-img map reports a hole based on BDRV_BLOCK_DATA, but nbd server reports a hole based on BDRV_BLOCK_ALLOCATED. The NBD protocol says: NBD_STATE_HOLE (bit 0): if set, the block represents a hole (and future writes to that area may cause fragmentation or encounter an NBD_ENOSPC error); if clear, the block is allocated or the server could not otherwise determine its status. qemu-img manual says: whether the sectors contain actual data or not (boolean field data; if false, the sectors are either unallocated or stored as optimized all-zero clusters); To me, data=false looks compatible with NBD_STATE_HOLE. From user point of view, getting same results from qemu-nbd and qemu-img is more important than being more correct about allocation status. Changing nbd server to report holes using BDRV_BLOCK_DATA makes qemu-nbd results compatible with qemu-img map: $ qemu-img map --output json nbd://localhost [{ "start": 0, "length": 6442450944, "depth": 0, "zero": true, "data": false, "offset": 0}] Signed-off-by: Nir Soffer Message-Id: <20210219160752.1826830-1-nsoffer@redhat.com> Reviewed-by: Eric Blake Reviewed-by: Vladimir Sementsov-Ogievskiy Signed-off-by: Eric Blake --- tests/qemu-iotests/241.out | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) (limited to 'tests/qemu-iotests') diff --git a/tests/qemu-iotests/241.out b/tests/qemu-iotests/241.out index 75f9f46..3f8c173 100644 --- a/tests/qemu-iotests/241.out +++ b/tests/qemu-iotests/241.out @@ -5,7 +5,7 @@ QA output created by 241 size: 1024 min block: 1 [{ "start": 0, "length": 1000, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, -{ "start": 1000, "length": 24, "depth": 0, "zero": true, "data": true, "offset": OFFSET}] +{ "start": 1000, "length": 24, "depth": 0, "zero": true, "data": false, "offset": OFFSET}] 1 KiB (0x400) bytes allocated at offset 0 bytes (0x0) === Exporting unaligned raw image, forced server sector alignment === @@ -23,6 +23,6 @@ WARNING: Image format was not specified for 'TEST_DIR/t.raw' and probing guessed size: 1024 min block: 1 [{ "start": 0, "length": 1000, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, -{ "start": 1000, "length": 24, "depth": 0, "zero": true, "data": true, "offset": OFFSET}] +{ "start": 1000, "length": 24, "depth": 0, "zero": true, "data": false, "offset": OFFSET}] 1 KiB (0x400) bytes allocated at offset 0 bytes (0x0) *** done -- cgit v1.1 From cf923b783efd565787e9ab006fb5608bb2a7297b Mon Sep 17 00:00:00 2001 From: Eric Blake Date: Thu, 11 Feb 2021 14:44:36 -0600 Subject: utils: Improve qemu_strtosz() to have 64 bits of precision MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit We have multiple clients of qemu_strtosz (qemu-io, the opts visitor, the keyval visitor), and it gets annoying that edge-case testing is impacted by implicit rounding to 53 bits of precision due to parsing with strtod(). As an example posted by Rich Jones: $ nbdkit memory $(( 2**63 - 2**30 )) --run \ 'build/qemu-io -f raw "$uri" -c "w -P 3 $(( 2**63 - 2**30 - 512 )) 512" ' write failed: Input/output error because 9223372035781033472 got rounded to 0x7fffffffc0000000 which is out of bounds. It is also worth noting that our existing parser, by virtue of using strtod(), accepts decimal AND hex numbers, even though test-cutils previously lacked any coverage of the latter until the previous patch. We do have existing clients that expect a hex parse to work (for example, iotest 33 using qemu-io -c "write -P 0xa 0x200 0x400"), but strtod() parses "08" as 8 rather than as an invalid octal number, so we know there are no clients that depend on octal. Our use of strtod() also means that "0x1.8k" would actually parse as 1536 (the fraction is 8/16), rather than 1843 (if the fraction were 8/10); but as this was not covered in the testsuite, I have no qualms forbidding hex fractions as invalid, so this patch declares that the use of fractions is only supported with decimal input, and enhances the testsuite to document that. Our previous use of strtod() meant that -1 parsed as a negative; now that we parse with strtoull(), negative values can wrap around modulo 2^64, so we have to explicitly check whether the user passed in a '-'; and make it consistent to also reject '-0'. This has the minor effect of treating negative values as EINVAL (with no change to endptr) rather than ERANGE (with endptr advanced to what was parsed), visible in the updated iotest output. We also had no testsuite coverage of "1.1e0k", which happened to parse under strtod() but is unlikely to occur in practice; as long as we are making things more robust, it is easy enough to reject the use of exponents in a strtod parse. The fix is done by breaking the parse into an integer prefix (no loss in precision), rejecting negative values (since we can no longer rely on strtod() to do that), determining if a decimal or hexadecimal parse was intended (with the new restriction that a fractional hex parse is not allowed), and where appropriate, using a floating point fractional parse (where we also scan to reject use of exponents in the fraction). The bulk of the patch is then updates to the testsuite to match our new precision, as well as adding new cases we reject (whether they were rejected or inadvertently accepted before). Signed-off-by: Eric Blake Message-Id: <20210211204438.1184395-3-eblake@redhat.com> Reviewed-by: Daniel P. Berrangé --- tests/qemu-iotests/049.out | 14 ++++++++++---- tests/qemu-iotests/178.out.qcow2 | 3 ++- tests/qemu-iotests/178.out.raw | 3 ++- 3 files changed, 14 insertions(+), 6 deletions(-) (limited to 'tests/qemu-iotests') diff --git a/tests/qemu-iotests/049.out b/tests/qemu-iotests/049.out index b1d8fd9..01f7b1f 100644 --- a/tests/qemu-iotests/049.out +++ b/tests/qemu-iotests/049.out @@ -92,16 +92,22 @@ Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off comp == 3. Invalid sizes == qemu-img create -f qcow2 TEST_DIR/t.qcow2 -- -1024 -qemu-img: Invalid image size specified. Must be between 0 and 9223372036854775807. +qemu-img: Invalid image size specified. You may use k, M, G, T, P or E suffixes for +qemu-img: kilobytes, megabytes, gigabytes, terabytes, petabytes and exabytes. qemu-img create -f qcow2 -o size=-1024 TEST_DIR/t.qcow2 -qemu-img: TEST_DIR/t.qcow2: Value '-1024' is out of range for parameter 'size' +qemu-img: TEST_DIR/t.qcow2: Parameter 'size' expects a non-negative number below 2^64 +Optional suffix k, M, G, T, P or E means kilo-, mega-, giga-, tera-, peta- +and exabytes, respectively. qemu-img create -f qcow2 TEST_DIR/t.qcow2 -- -1k -qemu-img: Invalid image size specified. Must be between 0 and 9223372036854775807. +qemu-img: Invalid image size specified. You may use k, M, G, T, P or E suffixes for +qemu-img: kilobytes, megabytes, gigabytes, terabytes, petabytes and exabytes. qemu-img create -f qcow2 -o size=-1k TEST_DIR/t.qcow2 -qemu-img: TEST_DIR/t.qcow2: Value '-1k' is out of range for parameter 'size' +qemu-img: TEST_DIR/t.qcow2: Parameter 'size' expects a non-negative number below 2^64 +Optional suffix k, M, G, T, P or E means kilo-, mega-, giga-, tera-, peta- +and exabytes, respectively. qemu-img create -f qcow2 TEST_DIR/t.qcow2 -- 1kilobyte qemu-img: Invalid image size specified. You may use k, M, G, T, P or E suffixes for diff --git a/tests/qemu-iotests/178.out.qcow2 b/tests/qemu-iotests/178.out.qcow2 index fe193fd..0d51fe4 100644 --- a/tests/qemu-iotests/178.out.qcow2 +++ b/tests/qemu-iotests/178.out.qcow2 @@ -13,7 +13,8 @@ qemu-img: Invalid option list: , qemu-img: Invalid parameter 'snapshot.foo' qemu-img: Failed in parsing snapshot param 'snapshot.foo=bar' qemu-img: --output must be used with human or json as argument. -qemu-img: Invalid image size specified. Must be between 0 and 9223372036854775807. +qemu-img: Invalid image size specified. You may use k, M, G, T, P or E suffixes for +qemu-img: kilobytes, megabytes, gigabytes, terabytes, petabytes and exabytes. qemu-img: Unknown file format 'foo' == Size calculation for a new file (human) == diff --git a/tests/qemu-iotests/178.out.raw b/tests/qemu-iotests/178.out.raw index 445e460..116241d 100644 --- a/tests/qemu-iotests/178.out.raw +++ b/tests/qemu-iotests/178.out.raw @@ -13,7 +13,8 @@ qemu-img: Invalid option list: , qemu-img: Invalid parameter 'snapshot.foo' qemu-img: Failed in parsing snapshot param 'snapshot.foo=bar' qemu-img: --output must be used with human or json as argument. -qemu-img: Invalid image size specified. Must be between 0 and 9223372036854775807. +qemu-img: Invalid image size specified. You may use k, M, G, T, P or E suffixes for +qemu-img: kilobytes, megabytes, gigabytes, terabytes, petabytes and exabytes. qemu-img: Unknown file format 'foo' == Size calculation for a new file (human) == -- cgit v1.1