Skip to content
GitLab
  1. Jun 22, 2022
    • Zqiang's avatar
      refscale: Convert test_lock spinlock to raw_spinlock · 7bf336fb
      Zqiang authored 
      

In kernels built with CONFIG_PREEMPT_RT=y, spinlocks are replaced by
rt_mutex, which can sleep.  This means that acquiring a non-raw spinlock
in a critical section where preemption is disabled can trigger the
following BUG:

BUG: scheduling while atomic: ref_scale_reade/76/0x00000002
Preemption disabled at:
ref_lock_section+0x16/0x80
Call Trace:
<TASK>
dump_stack_lvl+0x5b/0x82
dump_stack+0x10/0x12
__schedule_bug.cold+0x9c/0xad
__schedule+0x839/0xc00
schedule_rtlock+0x22/0x40
rtlock_slowlock_locked+0x460/0x1350
rt_spin_lock+0x61/0xe0
ref_lock_section+0x29/0x80
rcu_scale_one_reader+0x52/0x60
ref_scale_reader+0x28d/0x490
kthread+0x128/0x150
ret_from_fork+0x22/0x30
</TASK>

This commit therefore converts spinlock to raw_spinlock.

Signed-off-by: default avatarZqiang <qiang1.zhang@intel.com>
Signed-off-by: default avatarPaul E. McKenney <paulmck@kernel.org>
      7bf336fb
    • Li Qiong's avatar
      rcutorture: Handle failure of memory allocation functions · 1a5ca5e0
      Li Qiong authored 
      

This commit adds warnings for allocation failure during the mem_dump_obj()
tests.  It also terminates these tests upon such failure.

Signed-off-by: default avatarLi Qiong <liqiong@nfschina.com>
Signed-off-by: default avatarPaul E. McKenney <paulmck@kernel.org>
      1a5ca5e0
    • Frederic Weisbecker's avatar
      rcutorture: Fix ksoftirqd boosting timing and iteration · 3002153a
      Frederic Weisbecker authored 
      The RCU priority boosting can fail in two situations:

1) If (nr_cpus= > maxcpus=), which means if the total number of CPUs
is higher than those brought online at boot, then torture_onoff() may
later bring up CPUs that weren't online on boot. Now since rcutorture
initialization only boosts the ksoftirqds of the CPUs that have been
set online on boot, the CPUs later set online by torture_onoff won't
benefit from the boost, making RCU priority boosting fail.

2) The ksoftirqd kthreads are boosted after the creation of
rcu_torture_boost() kthreads, which opens a window large enough for these
rcu_torture_boost() kthreads to wait (despite running at FIFO priority)
for ksoftirqds that are still running at SCHED_NORMAL priority.

The issues can trigger for example with:

	./kvm.sh --configs TREE01 --kconfig "CONFIG_RCU_BOOST=y"

	[   34.968561] rcu-torture: !!!
	[   34.968627] ------------[ cut here ]------------
	[   35.014054] WARNING: CPU: 4 PID: 114 at kernel/rcu/rcutorture.c:1979 rcu_torture_stats_print+0x5ad/0x610
	[   35.052043] Modules linked in:
	[   35.069138] CPU: 4 PID: 114 Comm: rcu_torture_sta Not tainted 5.18.0-rc1 #1
	[   35.096424] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.14.0-0-g155821a-rebuilt.opensuse.org 04/01/2014
	[   35.154570] RIP: 0010:rcu_torture_stats_print+0x5ad/0x610
	[   35.198527] Code: 63 1b 02 00 74 02 0f 0b 48 83 3d 35 63 1b 02 00 74 02 0f 0b 48 83 3d 21 63 1b 02 00 74 02 0f 0b 48 83 3d 0d 63 1b 02 00 74 02 <0f> 0b 83 eb 01 0f 8e ba fc ff ff 0f 0b e9 b3 fc ff f82
	[   37.251049] RSP: 0000:ffffa92a0050bdf8 EFLAGS: 00010202
	[   37.277320] rcu: De-offloading 8
	[   37.290367] RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0000000000000001
	[   37.290387] RDX: 0000000000000000 RSI: 00000000ffffbfff RDI: 00000000ffffffff
	[   37.290398] RBP: 000000000000007b R08: 0000000000000000 R09: c0000000ffffbfff
	[   37.290407] R10: 000000000000002a R11: ffffa92a0050bc18 R12: ffffa92a0050be20
	[   37.290417] R13: ffffa92a0050be78 R14: 0000000000000000 R15: 000000000001bea0
	[   37.290427] FS:  0000000000000000(0000) GS:ffff96045eb00000(0000) knlGS:0000000000000000
	[   37.290448] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
	[   37.290460] CR2: 0000000000000000 CR3: 000000001dc0c000 CR4: 00000000000006e0
	[   37.290470] Call Trace:
	[   37.295049]  <TASK>
	[   37.295065]  ? preempt_count_add+0x63/0x90
	[   37.295095]  ? _raw_spin_lock_irqsave+0x12/0x40
	[   37.295125]  ? rcu_torture_stats_print+0x610/0x610
	[   37.295143]  rcu_torture_stats+0x29/0x70
	[   37.295160]  kthread+0xe3/0x110
	[   37.295176]  ? kthread_complete_and_exit+0x20/0x20
	[   37.295193]  ret_from_fork+0x22/0x30
	[   37.295218]  </TASK>

Fix this with boosting the ksoftirqds kthreads from the boosting
hotplug callback itself and before the boosting kthreads are created.

Fixes: ea6d962e

 ("rcutorture: Judge RCU priority boosting on grace periods, not callbacks")
Signed-off-by: default avatarFrederic Weisbecker <frederic@kernel.org>
Signed-off-by: default avatarPaul E. McKenney <paulmck@kernel.org>
      3002153a
    • Paul E. McKenney's avatar
      torture: Create kvm-check-branches.sh output in proper location · 148df92f
      Paul E. McKenney authored 
      

Currently, kvm-check-branches.sh causes each kvm.sh invocation create a
separate date-stamped directory, then after that invocation completes,
moves it into the *-group/NNNN directory.  This works, but makes it more
difficult to monitor an ongoing run.  This commit therefore uses the
kvm.sh --datestamp argument to make kvm.sh put the output in the right
place to start with, and also dispenses with the additional level of
datestamping.  (Those wanting datestamps can find them in the log files.)

Signed-off-by: default avatarPaul E. McKenney <paulmck@kernel.org>
      148df92f
    • Zqiang's avatar
      rcuscale: Fix smp_processor_id()-in-preemptible warnings · 92366810
      Zqiang authored 
      

Systems built with CONFIG_DEBUG_PREEMPT=y can trigger the following
BUG while running the rcuscale performance test:

BUG: using smp_processor_id() in preemptible [00000000] code: rcu_scale_write/69
CPU: 0 PID: 66 Comm: rcu_scale_write Not tainted 5.18.0-rc7-next-20220517-yoctodev-standard+
caller is debug_smp_processor_id+0x17/0x20
Call Trace:
<TASK>
dump_stack_lvl+0x49/0x5e
dump_stack+0x10/0x12
check_preemption_disabled+0xdf/0xf0
debug_smp_processor_id+0x17/0x20
rcu_scale_writer+0x2b5/0x580
kthread+0x177/0x1b0
ret_from_fork+0x22/0x30
</TASK>

Reproduction method:
runqemu kvm slirp nographic qemuparams="-m 4096 -smp 8" bootparams="isolcpus=2,3
nohz_full=2,3 rcu_nocbs=2,3 rcutree.dump_tree=1 rcuscale.shutdown=false
rcuscale.gp_async=true" -d

The problem is that the rcu_scale_writer() kthreads fail to set the
PF_NO_SETAFFINITY flags, which causes is_percpu_thread() to assume
that the kthread's affinity might change at any time, thus the BUG
noted above.

This commit therefore causes rcu_scale_writer() to set PF_NO_SETAFFINITY
in its kthread's ->flags field, thus preventing this BUG.

Signed-off-by: default avatarZqiang <qiang1.zhang@intel.com>
Signed-off-by: default avatarPaul E. McKenney <paulmck@kernel.org>
      92366810
    • Paul E. McKenney's avatar
      rcutorture: Make failure indication note reader-batch overflow · 8c0666d3
      Paul E. McKenney authored 
      

The loop scanning the pipesummary[] array currently skips the last
element, which means that the diagnostics ignore those rarest of
situations, namely where some readers persist across more than ten
grace periods, but all other readers avoid spanning a full grace period.
This commit therefore adjusts the scan to include the last element of
this array.

Signed-off-by: default avatarPaul E. McKenney <paulmck@kernel.org>
      8c0666d3
    • Paul E. McKenney's avatar
      torture: Adjust to again produce debugging information · 5c92d750
      Paul E. McKenney authored 
      A recent change to the DEBUG_INFO Kconfig option means that simply adding
CONFIG_DEBUG_INFO=y to the .config file and running "make oldconfig" no
longer works.  It is instead necessary to add CONFIG_DEBUG_INFO_NONE=n
and (for example) CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT=y.
This combination will then result in CONFIG_DEBUG_INFO being selected.

This commit therefore updates the Kconfig options produced in response
to the kvm.sh --gdb, --kasan, and --kcsan Kconfig options.

Fixes: f9b3cd24

 ("Kconfig.debug: make DEBUG_INFO selectable from a choice")
Signed-off-by: default avatarPaul E. McKenney <paulmck@kernel.org>
      5c92d750
    • Zqiang's avatar
      rcutorture: Fix memory leak in rcu_test_debug_objects() · 98ea2032
      Zqiang authored 
      

The kernel memory leak detector located the following:

unreferenced object 0xffff95d941135b50 (size 16):
  comm "swapper/0", pid 1, jiffies 4294667610 (age 1367.451s)
  hex dump (first 16 bytes):
    f0 c6 c2 bd d9 95 ff ff 00 00 00 00 00 00 00 00  ................
  backtrace:
    [<00000000bc81d9b1>] kmem_cache_alloc_trace+0x2f6/0x500
    [<00000000d28be229>] rcu_torture_init+0x1235/0x1354
    [<0000000032c3acd9>] do_one_initcall+0x51/0x210
    [<000000003c117727>] kernel_init_freeable+0x205/0x259
    [<000000003961f965>] kernel_init+0x1a/0x120
    [<000000001998f890>] ret_from_fork+0x22/0x30

This is caused by the rcu_test_debug_objects() function allocating an
rcu_head structure, then failing to free it.  This commit therefore adds
the needed kfree() after the last use of this structure.

Signed-off-by: default avatarZqiang <qiang1.zhang@intel.com>
Signed-off-by: default avatarPaul E. McKenney <paulmck@kernel.org>
      98ea2032
    • Paul E. McKenney's avatar
      rcutorture: Simplify rcu_torture_read_exit_child() loop · d984114e
      Paul E. McKenney authored 
      

The existing loop has an implicit manual loop that obscures the flow
and requires an extra control variable.  This commit makes this implicit
loop explicit, thus saving several lines of code.

Signed-off-by: default avatarPaul E. McKenney <paulmck@kernel.org>
      d984114e
    • Anna-Maria Behnsen's avatar
      rcu/torture: Change order of warning and trace dump · 14c0017c
      Anna-Maria Behnsen authored 
      

Dumping a big ftrace buffer could lead to a RCU stall. So there is the
ftrace buffer and the stall information which needs to be printed. When
there is additionally a WARN_ON() which describes the reason for the ftrace
buffer dump and the WARN_ON() is executed _after_ ftrace buffer dump, the
information get lost in the middle of the RCU stall information.

Therefore print WARN_ON() message before dumping the ftrace buffer in
rcu_torture_writer().

[ paulmck: Add tracing_off() to avoid cruft from WARN(). ]

Signed-off-by: default avatarAnna-Maria Behnsen <anna-maria@linutronix.de>
Reviewed-by: default avatarBenedikt Spranger <b.spranger@linutronix.de>
Signed-off-by: default avatarPaul E. McKenney <paulmck@kernel.org>
      14c0017c
  3. Jun 21, 2022
    • Paul E. McKenney's avatar
      torture: Make kvm-remote.sh announce which system is being waited on · ab69d3c8
      Paul E. McKenney authored 
      

If a remote system fails in certain ways, for example, if it is rebooted
without removing the contents of the /tmp directory, its remote.run file
never will be removed and the kvm-remote.sh script will loop waiting
forever.  The manual workaround for this (hopefully!) rare event is to
manually remove the file, which will cause the results up to the reboot
to be collected and evaluated.

Unfortunately, to work out which system is holding things up, the user
must refer to the name of the last system whose results were collected,
then look up the name of the next system in sequence, then manually
remove the remote.run file.  Even more unfortunately, this procedure can
be fooled in runs where each system handles more than one batch should
a given system take longer than expected, causing the systems to be
handled out of order.

This commit therefore causes kvm-remote.sh to print out the name of
the system it will wait on next, allowing the user to refer directly
to that name.  Making the kvm-remote.sh script automatically handle
unscheduled termination of the qemu processes is left as future work.
Quite possibly deep future work.

Signed-off-by: default avatarPaul E. McKenney <paulmck@kernel.org>
      ab69d3c8
  5. Jun 20, 2022
  7. Jun 19, 2022
  9. Jun 18, 2022