diff options
author | Paolo Bonzini <pbonzini@redhat.com> | 2020-04-07 10:07:46 -0400 |
---|---|---|
committer | Stefan Hajnoczi <stefanha@redhat.com> | 2020-04-09 16:17:14 +0100 |
commit | 5710a3e09f9b85801e5ce70797a4a511e5fc9e2c (patch) | |
tree | a652eb8d95fe90b4e9ba8056ca5afadec87f89dc /util/aio-posix.c | |
parent | 3c18a92dc4b55ca8cc37a755ed119f11c0f34099 (diff) | |
download | qemu-5710a3e09f9b85801e5ce70797a4a511e5fc9e2c.zip qemu-5710a3e09f9b85801e5ce70797a4a511e5fc9e2c.tar.gz qemu-5710a3e09f9b85801e5ce70797a4a511e5fc9e2c.tar.bz2 |
async: use explicit memory barriers
When using C11 atomics, non-seqcst reads and writes do not participate
in the total order of seqcst operations. In util/async.c and util/aio-posix.c,
in particular, the pattern that we use
write ctx->notify_me write bh->scheduled
read bh->scheduled read ctx->notify_me
if !bh->scheduled, sleep if ctx->notify_me, notify
needs to use seqcst operations for both the write and the read. In
general this is something that we do not want, because there can be
many sources that are polled in addition to bottom halves. The
alternative is to place a seqcst memory barrier between the write
and the read. This also comes with a disadvantage, in that the
memory barrier is implicit on strongly-ordered architectures and
it wastes a few dozen clock cycles.
Fortunately, ctx->notify_me is never written concurrently by two
threads, so we can assert that and relax the writes to ctx->notify_me.
The resulting solution works and performs well on both aarch64 and x86.
Note that the atomic_set/atomic_read combination is not an atomic
read-modify-write, and therefore it is even weaker than C11 ATOMIC_RELAXED;
on x86, ATOMIC_RELAXED compiles to a locked operation.
Analyzed-by: Ying Fang <fangying1@huawei.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Tested-by: Ying Fang <fangying1@huawei.com>
Message-Id: <20200407140746.8041-6-pbonzini@redhat.com>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Diffstat (limited to 'util/aio-posix.c')
-rw-r--r-- | util/aio-posix.c | 16 |
1 files changed, 14 insertions, 2 deletions
diff --git a/util/aio-posix.c b/util/aio-posix.c index cd6cf0a..c3613d2 100644 --- a/util/aio-posix.c +++ b/util/aio-posix.c @@ -559,6 +559,11 @@ bool aio_poll(AioContext *ctx, bool blocking) int64_t timeout; int64_t start = 0; + /* + * There cannot be two concurrent aio_poll calls for the same AioContext (or + * an aio_poll concurrent with a GSource prepare/check/dispatch callback). + * We rely on this below to avoid slow locked accesses to ctx->notify_me. + */ assert(in_aio_context_home_thread(ctx)); /* aio_notify can avoid the expensive event_notifier_set if @@ -569,7 +574,13 @@ bool aio_poll(AioContext *ctx, bool blocking) * so disable the optimization now. */ if (blocking) { - atomic_add(&ctx->notify_me, 2); + atomic_set(&ctx->notify_me, atomic_read(&ctx->notify_me) + 2); + /* + * Write ctx->notify_me before computing the timeout + * (reading bottom half flags, etc.). Pairs with + * smp_mb in aio_notify(). + */ + smp_mb(); } qemu_lockcnt_inc(&ctx->list_lock); @@ -590,7 +601,8 @@ bool aio_poll(AioContext *ctx, bool blocking) } if (blocking) { - atomic_sub(&ctx->notify_me, 2); + /* Finish the poll before clearing the flag. */ + atomic_store_release(&ctx->notify_me, atomic_read(&ctx->notify_me) - 2); aio_notify_accept(ctx); } |