1. Jun 22, 2022
    • Paul E. McKenney's avatar
      rcu-tasks: Eliminate RCU Tasks Trace IPIs to online CPUs · e386b672
      Paul E. McKenney authored
      
      
      Currently, the RCU Tasks Trace grace-period kthread IPIs each online CPU
      using smp_call_function_single() in order to track any tasks currently in
      RCU Tasks Trace read-side critical sections during which the corresponding
      task has neither blocked nor been preempted.  These IPIs are annoying
      and are also not strictly necessary because any task that blocks or is
      preempted within its current RCU Tasks Trace read-side critical section
      will be tracked on one of the per-CPU rcu_tasks_percpu structure's
      ->rtp_blkd_tasks list.  So the only time that this is a problem is if
      one of the CPUs runs through a long-duration RCU Tasks Trace read-side
      critical section without a context switch.
      
      Note that the task_call_func() function cannot help here because there is
      no safe way to identify the target task.  Of course, the task_call_func()
      function will be very useful later, when processing the list of tasks,
      but it needs to know the task.
      
      This commit therefore creates a cpu_curr_snapshot() function that returns
      a pointer the task_struct structure of some task that happened to be
      running on the specified CPU more or less during the time that the
      cpu_curr_snapshot() function was executing.  If there was no context
      switch during this time, this function will return a pointer to the
      task_struct structure of the task that was running throughout.  If there
      was a context switch, then the outgoing task will be taken care of by
      RCU's context-switch hook, and the incoming task was either already taken
      care during some previous context switch, or it is not currently within an
      RCU Tasks Trace read-side critical section.  And in this latter case, the
      grace period already started, so there is no need to wait on this task.
      
      This new cpu_curr_snapshot() function is invoked on each CPU early in
      the RCU Tasks Trace grace-period processing, and the resulting tasks
      are queued for later quiescent-state inspection.
      
      Signed-off-by: default avatarPaul E. McKenney <paulmck@kernel.org>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: Neeraj Upadhyay <quic_neeraju@quicinc.com>
      Cc: Eric Dumazet <edumazet@google.com>
      Cc: Alexei Starovoitov <ast@kernel.org>
      Cc: Andrii Nakryiko <andrii@kernel.org>
      Cc: Martin KaFai Lau <kafai@fb.com>
      Cc: KP Singh <kpsingh@kernel.org>
      e386b672
    • Paul E. McKenney's avatar
      rcu-tasks: Maintain a count of tasks blocking RCU Tasks Trace grace period · ffcc21a3
      Paul E. McKenney authored
      
      
      This commit maintains a new n_trc_holdouts counter that tracks the number
      of tasks blocking the RCU Tasks grace period.  This counter is useful
      for debugging, and its value has been added to a diagostic message.
      
      Signed-off-by: default avatarPaul E. McKenney <paulmck@kernel.org>
      Cc: Neeraj Upadhyay <quic_neeraju@quicinc.com>
      Cc: Eric Dumazet <edumazet@google.com>
      Cc: Alexei Starovoitov <ast@kernel.org>
      Cc: Andrii Nakryiko <andrii@kernel.org>
      Cc: Martin KaFai Lau <kafai@fb.com>
      Cc: KP Singh <kpsingh@kernel.org>
      ffcc21a3
    • Paul E. McKenney's avatar
      rcu-tasks: Stop RCU Tasks Trace from scanning full tasks list · 1a4a8153
      Paul E. McKenney authored
      
      
      This commit takes off the training wheels and relies only on scanning
      currently running tasks and tasks that have blocked or been preempted
      within their current RCU Tasks Trace read-side critical section.
      
      Before this commit, the time complexity of an RCU Tasks Trace grace
      period is O(T), where T is the number of tasks.  After this commit,
      this time complexity is O(C+B), where C is the number of CPUs and B
      is the number of tasks that have blocked (or been preempted) at least
      once during their current RCU Tasks Trace read-side critical sections.
      Of course, if all tasks have blocked (or been preempted) at least once
      during their current RCU Tasks Trace read-side critical sections, this is
      still O(T), but current expectations are that RCU Tasks Trace read-side
      critical section will be short and that there will normally not be large
      numbers of tasks blocked within such a critical section.
      
      Dave Marchevsky kindly measured the effects of this commit on the RCU
      Tasks Trace grace-period latency and the rcu_tasks_trace_kthread task's
      CPU consumption per RCU Tasks Trace grace period over the course of a
      fixed test, all in milliseconds:
      
      		Before			After
      
      GP latency	22.3 ms stddev > 0.1	17.0 ms stddev < 0.1
      
      GP CPU		 2.3 ms stddev 0.3	 1.1 ms stddev 0.2
      
      This was on a system with 15,000 tasks, so it is reasonable to expect
      much larger savings on the systems on which this issue was first noted,
      given that they sport well in excess of 100,000 tasks.  CPU consumption
      was measured using profiling techniques.
      
      Signed-off-by: default avatarPaul E. McKenney <paulmck@kernel.org>
      Cc: Neeraj Upadhyay <quic_neeraju@quicinc.com>
      Cc: Eric Dumazet <edumazet@google.com>
      Cc: Alexei Starovoitov <ast@kernel.org>
      Cc: Andrii Nakryiko <andrii@kernel.org>
      Cc: Martin KaFai Lau <kafai@fb.com>
      Cc: KP Singh <kpsingh@kernel.org>
      Tested-by: default avatarDave Marchevsky <davemarchevsky@fb.com>
      1a4a8153
  2. Jun 21, 2022
    • Paul E. McKenney's avatar
      rcu-tasks: Stop RCU Tasks Trace from scanning idle tasks · 955a0192
      Paul E. McKenney authored
      
      
      Now that RCU scans both running tasks and tasks that have blocked within
      their current RCU Tasks Trace read-side critical section, there is no
      need for it to scan the idle tasks.  After all, an idle loop should not
      be remain within an RCU Tasks Trace read-side critical section across
      exit from idle, and from a BPF viewpoint, functions invoked from the
      idle loop should not sleep.  So only running idle tasks can be within
      RCU Tasks Trace read-side critical sections.
      
      This commit therefore removes the scan of the idle tasks from the
      rcu_tasks_trace_postscan() function.
      
      Signed-off-by: default avatarPaul E. McKenney <paulmck@kernel.org>
      Cc: Neeraj Upadhyay <quic_neeraju@quicinc.com>
      Cc: Eric Dumazet <edumazet@google.com>
      Cc: Alexei Starovoitov <ast@kernel.org>
      Cc: Andrii Nakryiko <andrii@kernel.org>
      Cc: Martin KaFai Lau <kafai@fb.com>
      Cc: KP Singh <kpsingh@kernel.org>
      955a0192
    • Paul E. McKenney's avatar
      rcu-tasks: Pull in tasks blocked within RCU Tasks Trace readers · dc7d54b4
      Paul E. McKenney authored
      
      
      This commit scans each CPU's ->rtp_blkd_tasks list, adding them to
      the list of holdout tasks.  This will cause the current RCU Tasks Trace
      grace period to wait until these tasks exit their RCU Tasks Trace
      read-side critical sections.  This commit will enable later work
      omitting the scan of the full task list.
      
      Signed-off-by: default avatarPaul E. McKenney <paulmck@kernel.org>
      Cc: Neeraj Upadhyay <quic_neeraju@quicinc.com>
      Cc: Eric Dumazet <edumazet@google.com>
      Cc: Alexei Starovoitov <ast@kernel.org>
      Cc: Andrii Nakryiko <andrii@kernel.org>
      Cc: Martin KaFai Lau <kafai@fb.com>
      Cc: KP Singh <kpsingh@kernel.org>
      dc7d54b4
    • Paul E. McKenney's avatar
      rcu-tasks: Scan running tasks for RCU Tasks Trace readers · 7460ade1
      Paul E. McKenney authored
      
      
      A running task might be within an RCU Tasks Trace read-side critical
      section for any length of time, but will not be placed on any of the
      per-CPU rcu_tasks_percpu structure's ->rtp_blkd_tasks lists.  Therefore
      any RCU Tasks Trace grace-period processing that does not scan the full
      task list must interact with the running tasks.
      
      This commit therefore causes the rcu_tasks_trace_pregp_step() function
      to IPI each CPU in order to place the corresponding task on the holdouts
      list and to record whether or not it was in an RCU Tasks Trace read-side
      critical section.  Yes, it is possible to avoid adding it to that list
      if it is not a reader, but that would prevent the system from remembering
      that this task was in a quiescent state.  Which is why the running tasks
      are unconditionally added to the holdout list.
      
      Signed-off-by: default avatarPaul E. McKenney <paulmck@kernel.org>
      Cc: Neeraj Upadhyay <quic_neeraju@quicinc.com>
      Cc: Eric Dumazet <edumazet@google.com>
      Cc: Alexei Starovoitov <ast@kernel.org>
      Cc: Andrii Nakryiko <andrii@kernel.org>
      Cc: Martin KaFai Lau <kafai@fb.com>
      Cc: KP Singh <kpsingh@kernel.org>
      7460ade1
    • Paul E. McKenney's avatar
      rcu-tasks: Avoid rcu_tasks_trace_pertask() duplicate list additions · 19415004
      Paul E. McKenney authored
      
      
      This commit adds checks within rcu_tasks_trace_pertask() to avoid
      duplicate (and destructive) additions to the holdouts list.  These checks
      will be required later due to the possibility of a given task having
      blocked while in an RCU Tasks Trace read-side critical section, but now
      running on a CPU.
      
      Signed-off-by: default avatarPaul E. McKenney <paulmck@kernel.org>
      Cc: Neeraj Upadhyay <quic_neeraju@quicinc.com>
      Cc: Eric Dumazet <edumazet@google.com>
      Cc: Alexei Starovoitov <ast@kernel.org>
      Cc: Andrii Nakryiko <andrii@kernel.org>
      Cc: Martin KaFai Lau <kafai@fb.com>
      Cc: KP Singh <kpsingh@kernel.org>
      19415004
    • Paul E. McKenney's avatar
      rcu-tasks: Move rcu_tasks_trace_pertask() before rcu_tasks_trace_pregp_step() · 1fa98e2e
      Paul E. McKenney authored
      
      
      This is a code-motion-only commit that moves rcu_tasks_trace_pertask()
      to precede rcu_tasks_trace_pregp_step(), so that the latter will be
      able to invoke the other without forward references.
      
      Signed-off-by: default avatarPaul E. McKenney <paulmck@kernel.org>
      Cc: Neeraj Upadhyay <quic_neeraju@quicinc.com>
      Cc: Eric Dumazet <edumazet@google.com>
      Cc: Alexei Starovoitov <ast@kernel.org>
      Cc: Andrii Nakryiko <andrii@kernel.org>
      Cc: Martin KaFai Lau <kafai@fb.com>
      Cc: KP Singh <kpsingh@kernel.org>
      1fa98e2e
    • Paul E. McKenney's avatar
      rcu-tasks: Add blocked-task indicator to RCU Tasks Trace stall warnings · 387c0ad7
      Paul E. McKenney authored
      
      
      This commit adds a "B" indicator to the RCU Tasks Trace CPU stall warning
      when the task has blocked within its current read-side critical section.
      This serves as a debugging aid.
      
      Signed-off-by: default avatarPaul E. McKenney <paulmck@kernel.org>
      Cc: Neeraj Upadhyay <quic_neeraju@quicinc.com>
      Cc: Eric Dumazet <edumazet@google.com>
      Cc: Alexei Starovoitov <ast@kernel.org>
      Cc: Andrii Nakryiko <andrii@kernel.org>
      Cc: Martin KaFai Lau <kafai@fb.com>
      Cc: KP Singh <kpsingh@kernel.org>
      387c0ad7
    • Paul E. McKenney's avatar
      rcu-tasks: Untrack blocked RCU Tasks Trace at reader end · 0bcb3868
      Paul E. McKenney authored
      
      
      This commit causes rcu_read_unlock_trace() to check for the current
      task being on a per-CPU list within the rcu_tasks_percpu structure,
      and removes it from that list if so.  This has the effect of curtailing
      tracking of a task that blocked within an RCU Tasks Trace read-side
      critical section once it exits that critical section.
      
      Signed-off-by: default avatarPaul E. McKenney <paulmck@kernel.org>
      Cc: Neeraj Upadhyay <quic_neeraju@quicinc.com>
      Cc: Eric Dumazet <edumazet@google.com>
      Cc: Alexei Starovoitov <ast@kernel.org>
      Cc: Andrii Nakryiko <andrii@kernel.org>
      Cc: Martin KaFai Lau <kafai@fb.com>
      Cc: KP Singh <kpsingh@kernel.org>
      0bcb3868
    • Paul E. McKenney's avatar
      rcu-tasks: Track blocked RCU Tasks Trace readers · 0356d4e6
      Paul E. McKenney authored
      
      
      This commit places any task that has ever blocked within its current
      RCU Tasks Trace read-side critical section on a per-CPU list within the
      rcu_tasks_percpu structure.  Tasks are removed from this list when they
      exit by the exit_tasks_rcu_finish_trace() function.  The purpose of this
      commit is to provide the information needed to eliminate the current
      scan of the full task list.
      
      This commit offsets the INT_MIN value for ->trc_reader_nesting with the
      new nesting level in order to avoid queueing tasks that are exiting
      their read-side critical sections.
      
      [ paulmck: Apply kernel test robot feedback. ]
      [ paulmck: Apply feedback from syzbot+9bb26e7c5e8e4fa7e641@syzkaller.appspotmail.com ]
      
      Signed-off-by: default avatarPaul E. McKenney <paulmck@kernel.org>
      Tested-by: default avatarsyzbot <syzbot+9bb26e7c5e8e4fa7e641@syzkaller.appspotmail.com>
      Tested-by: default avatar"Zhang, Qiang1" <qiang1.zhang@intel.com>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: Neeraj Upadhyay <quic_neeraju@quicinc.com>
      Cc: Eric Dumazet <edumazet@google.com>
      Cc: Alexei Starovoitov <ast@kernel.org>
      Cc: Andrii Nakryiko <andrii@kernel.org>
      Cc: Martin KaFai Lau <kafai@fb.com>
      Cc: KP Singh <kpsingh@kernel.org>
      0356d4e6
    • Paul E. McKenney's avatar
      rcu-tasks: Add data structures for lightweight grace periods · 434c9eef
      Paul E. McKenney authored
      
      
      This commit adds fields to task_struct and to rcu_tasks_percpu that will
      be used to avoid the task-list scan for RCU Tasks Trace grace periods,
      and also initializes these fields.
      
      Signed-off-by: default avatarPaul E. McKenney <paulmck@kernel.org>
      Cc: Neeraj Upadhyay <quic_neeraju@quicinc.com>
      Cc: Eric Dumazet <edumazet@google.com>
      Cc: Alexei Starovoitov <ast@kernel.org>
      Cc: Andrii Nakryiko <andrii@kernel.org>
      Cc: Martin KaFai Lau <kafai@fb.com>
      Cc: KP Singh <kpsingh@kernel.org>
      434c9eef
    • Paul E. McKenney's avatar
      rcu-tasks: Make RCU Tasks Trace stall warning handle idle offline tasks · f90f19da
      Paul E. McKenney authored
      
      
      When a CPU is offline, its idle task can appear to be running, but it
      cannot be doing anything while CPU-hotplug operations are excluded.
      This commit takes advantage of that fact by making trc_check_slow_task()
      check for task_curr(t) && cpu_online(task_cpu(t)), and recording
      full information in that case.
      
      Signed-off-by: default avatarPaul E. McKenney <paulmck@kernel.org>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: Neeraj Upadhyay <quic_neeraju@quicinc.com>
      Cc: Eric Dumazet <edumazet@google.com>
      Cc: Alexei Starovoitov <ast@kernel.org>
      Cc: Andrii Nakryiko <andrii@kernel.org>
      Cc: Martin KaFai Lau <kafai@fb.com>
      Cc: KP Singh <kpsingh@kernel.org>
      f90f19da
    • Paul E. McKenney's avatar
      rcu-tasks: Make RCU Tasks Trace stall warnings print full .b.need_qs field · be15a164
      Paul E. McKenney authored
      
      
      Currently, the RCU Tasks Trace CPU stall warning simply indicates
      whether or not the .b.need_qs field is zero.  This commit shows the
      three permitted values and flags other values with either "!" or "?".
      This is a debugging aid.
      
      Signed-off-by: default avatarPaul E. McKenney <paulmck@kernel.org>
      Cc: Neeraj Upadhyay <quic_neeraju@quicinc.com>
      Cc: Eric Dumazet <edumazet@google.com>
      Cc: Alexei Starovoitov <ast@kernel.org>
      Cc: Andrii Nakryiko <andrii@kernel.org>
      Cc: Martin KaFai Lau <kafai@fb.com>
      Cc: KP Singh <kpsingh@kernel.org>
      be15a164
    • Paul E. McKenney's avatar
      rcu-tasks: Flag offline CPUs in RCU Tasks Trace stall warnings · c8c03ad9
      Paul E. McKenney authored
      
      
      This commit tags offline CPUs with "(offline)" in RCU Tasks Trace CPU
      stall warnings.  This is a debugging aid.
      
      Signed-off-by: default avatarPaul E. McKenney <paulmck@kernel.org>
      Cc: Neeraj Upadhyay <quic_neeraju@quicinc.com>
      Cc: Eric Dumazet <edumazet@google.com>
      Cc: Alexei Starovoitov <ast@kernel.org>
      Cc: Andrii Nakryiko <andrii@kernel.org>
      Cc: Martin KaFai Lau <kafai@fb.com>
      Cc: KP Singh <kpsingh@kernel.org>
      c8c03ad9
    • Paul E. McKenney's avatar
      rcu-tasks: Add slow-IPI indicator to RCU Tasks Trace stall warnings · 9f3eb5fb
      Paul E. McKenney authored
      
      
      This commit adds a "I" indicator to the RCU Tasks Trace CPU stall
      warning when an IPI directed to a task has thus far failed to arrive.
      This serves as a debugging aid.
      
      Signed-off-by: default avatarPaul E. McKenney <paulmck@kernel.org>
      Cc: Neeraj Upadhyay <quic_neeraju@quicinc.com>
      Cc: Eric Dumazet <edumazet@google.com>
      Cc: Alexei Starovoitov <ast@kernel.org>
      Cc: Andrii Nakryiko <andrii@kernel.org>
      Cc: Martin KaFai Lau <kafai@fb.com>
      Cc: KP Singh <kpsingh@kernel.org>
      9f3eb5fb
    • Paul E. McKenney's avatar
      rcu-tasks: Simplify trc_inspect_reader() QS logic · 0968e892
      Paul E. McKenney authored
      
      
      Currently, trc_inspect_reader() does one check for nesting less than
      or equal to zero, then sorts out the distinctions within this single
      "if" statement.  This commit simplifies the logic by providing one
      "if" statement for quiescent states (nesting of zero) and another "if"
      statement for transitioning from one nesting level to another or the
      outermost rcu_read_unlock_trace() (negative nesting).
      
      Signed-off-by: default avatarPaul E. McKenney <paulmck@kernel.org>
      Cc: Neeraj Upadhyay <quic_neeraju@quicinc.com>
      Cc: Eric Dumazet <edumazet@google.com>
      Cc: Alexei Starovoitov <ast@kernel.org>
      Cc: Andrii Nakryiko <andrii@kernel.org>
      Cc: Martin KaFai Lau <kafai@fb.com>
      Cc: KP Singh <kpsingh@kernel.org>
      0968e892
    • Paul E. McKenney's avatar
      rcu-tasks: Make rcu_note_context_switch() unconditionally call rcu_tasks_qs() · 6a694411
      Paul E. McKenney authored
      
      
      This commit makes rcu_note_context_switch() unconditionally invoke the
      rcu_tasks_qs() function, as opposed to doing so only when RCU (as opposed
      to RCU Tasks Trace) urgently needs a grace period to end.
      
      Signed-off-by: default avatarPaul E. McKenney <paulmck@kernel.org>
      Cc: Neeraj Upadhyay <quic_neeraju@quicinc.com>
      Cc: Eric Dumazet <edumazet@google.com>
      Cc: Alexei Starovoitov <ast@kernel.org>
      Cc: Andrii Nakryiko <andrii@kernel.org>
      Cc: Martin KaFai Lau <kafai@fb.com>
      Cc: KP Singh <kpsingh@kernel.org>
      6a694411
    • Paul E. McKenney's avatar
      rcu-tasks: RCU Tasks Trace grace-period kthread has implicit QS · 5d4c90d7
      Paul E. McKenney authored
      
      
      Because the task driving the grace-period kthread is in quiescent state
      throughout, this commit excludes it from the list of tasks from which
      a quiescent state is needed.
      
      This does mean that attaching a sleepable BPF program to function in
      kernel/rcu/tasks.h is a bad idea, by the way.
      
      Signed-off-by: default avatarPaul E. McKenney <paulmck@kernel.org>
      Cc: Neeraj Upadhyay <quic_neeraju@quicinc.com>
      Cc: Eric Dumazet <edumazet@google.com>
      Cc: Alexei Starovoitov <ast@kernel.org>
      Cc: Andrii Nakryiko <andrii@kernel.org>
      Cc: Martin KaFai Lau <kafai@fb.com>
      Cc: KP Singh <kpsingh@kernel.org>
      5d4c90d7
    • Paul E. McKenney's avatar
      rcu-tasks: Handle idle tasks for recently offlined CPUs · 897ba84d
      Paul E. McKenney authored
      
      
      This commit identifies idle tasks for recently offlined CPUs as residing
      in a quiescent state.  This is safe only because CPU-hotplug operations
      are excluded during these checks.
      
      Signed-off-by: default avatarPaul E. McKenney <paulmck@kernel.org>
      Cc: Neeraj Upadhyay <quic_neeraju@quicinc.com>
      Cc: Eric Dumazet <edumazet@google.com>
      Cc: Alexei Starovoitov <ast@kernel.org>
      Cc: Andrii Nakryiko <andrii@kernel.org>
      Cc: Martin KaFai Lau <kafai@fb.com>
      Cc: KP Singh <kpsingh@kernel.org>
      897ba84d
    • Paul E. McKenney's avatar
      rcu-tasks: Idle tasks on offline CPUs are in quiescent states · 5c9a9ca4
      Paul E. McKenney authored
      
      
      Any idle task corresponding to an offline CPU is in an RCU Tasks Trace
      quiescent state.  This commit causes rcu_tasks_trace_postscan() to ignore
      idle tasks for offline CPUs, which it can do safely due to CPU-hotplug
      operations being disabled.
      
      Signed-off-by: default avatarPaul E. McKenney <paulmck@kernel.org>
      Cc: Neeraj Upadhyay <quic_neeraju@quicinc.com>
      Cc: Eric Dumazet <edumazet@google.com>
      Cc: Alexei Starovoitov <ast@kernel.org>
      Cc: Andrii Nakryiko <andrii@kernel.org>
      Cc: Martin KaFai Lau <kafai@fb.com>
      Cc: KP Singh <kpsingh@kernel.org>
      5c9a9ca4
    • Paul E. McKenney's avatar
      rcu-tasks: Make trc_read_check_handler() fetch ->trc_reader_nesting only once · 9ff86b4c
      Paul E. McKenney authored
      
      
      This commit replaces the pair of READ_ONCE(t->trc_reader_nesting) calls
      with a single such call and a local variable.  This makes the code's
      intent more clear.
      
      Signed-off-by: default avatarPaul E. McKenney <paulmck@kernel.org>
      Cc: Neeraj Upadhyay <quic_neeraju@quicinc.com>
      Cc: Eric Dumazet <edumazet@google.com>
      Cc: Alexei Starovoitov <ast@kernel.org>
      Cc: Andrii Nakryiko <andrii@kernel.org>
      Cc: Martin KaFai Lau <kafai@fb.com>
      Cc: KP Singh <kpsingh@kernel.org>
      9ff86b4c
    • Paul E. McKenney's avatar
      rcu-tasks: Remove rcu_tasks_trace_postgp() wait for counter · 55061126
      Paul E. McKenney authored
      
      
      Now that tasks are not removed from the list until they have responded to
      any needed request for a quiescent state, it is no longer necessary to
      wait for the trc_n_readers_need_end counter to go to zero.  This commit
      therefore removes that waiting code.
      
      It is therefore also no longer necessary for rcu_tasks_trace_postgp() to
      do the final decrement of this counter, so that code is also removed.
      This in turn means that trc_n_readers_need_end counter itself can
      be removed, as can the rcu_tasks_trace_iw irq_work structure and the
      rcu_read_unlock_iw() function.
      
      [ paulmck: Apply feedback from Zqiang. ]
      
      Signed-off-by: default avatarPaul E. McKenney <paulmck@kernel.org>
      Cc: Neeraj Upadhyay <quic_neeraju@quicinc.com>
      Cc: Eric Dumazet <edumazet@google.com>
      Cc: Alexei Starovoitov <ast@kernel.org>
      Cc: Andrii Nakryiko <andrii@kernel.org>
      Cc: Martin KaFai Lau <kafai@fb.com>
      Cc: KP Singh <kpsingh@kernel.org>
      55061126
    • Paul E. McKenney's avatar
      rcu-tasks: Merge state into .b.need_qs and atomically update · 3847b645
      Paul E. McKenney authored
      
      
      This commit gets rid of the task_struct structure's ->trc_reader_checked
      field, making it instead be a bit within the task_struct structure's
      existing ->trc_reader_special.b.need_qs field.  This commit also
      atomically loads, stores, and checks the resulting combination of the
      reader-checked and need-quiescent state flags.  This will in turn allow
      significant simplification of the rcu_tasks_trace_postgp() function
      as well as elimination of the trc_n_readers_need_end counter in later
      commits.  These changes will in turn simplify later elimination of the
      RCU Tasks Trace scan of the task list, which will make RCU Tasks Trace
      grace periods less CPU-intensive.
      
      [ paulmck: Apply kernel test robot feedback. ]
      
      Signed-off-by: default avatarPaul E. McKenney <paulmck@kernel.org>
      Cc: Neeraj Upadhyay <quic_neeraju@quicinc.com>
      Cc: Eric Dumazet <edumazet@google.com>
      Cc: Alexei Starovoitov <ast@kernel.org>
      Cc: Andrii Nakryiko <andrii@kernel.org>
      Cc: Martin KaFai Lau <kafai@fb.com>
      Cc: KP Singh <kpsingh@kernel.org>
      3847b645
    • Paul E. McKenney's avatar
      rcu-tasks: Drive synchronous grace periods from calling task · 4a8cc433
      Paul E. McKenney authored
      
      
      This commit causes synchronous grace periods to be driven from the task
      invoking synchronize_rcu_*(), allowing these functions to be invoked from
      the mid-boot dead zone extending from when the scheduler was initialized
      to to point that the various RCU tasks grace-period kthreads are spawned.
      This change will allow the self-tests to run in a consistent manner.
      
      Reported-by: default avatarMatthew Wilcox <willy@infradead.org>
      Reported-by: default avatarZhouyi Zhou <zhouzhouyi@gmail.com>
      Signed-off-by: default avatarPaul E. McKenney <paulmck@kernel.org>
      4a8cc433
    • Paul E. McKenney's avatar
      rcu-tasks: Move synchronize_rcu_tasks_generic() down · 68cb4720
      Paul E. McKenney authored
      
      
      This is strictly a code-motion commit that moves the
      synchronize_rcu_tasks_generic() down to where it can invoke
      rcu_tasks_one_gp() without the need for a forward declaration.
      
      Signed-off-by: default avatarPaul E. McKenney <paulmck@kernel.org>
      68cb4720
    • Paul E. McKenney's avatar
      rcu-tasks: Split rcu_tasks_one_gp() from rcu_tasks_kthread() · d96225fd
      Paul E. McKenney authored
      
      
      This commit abstracts most of the rcu_tasks_kthread() function's loop
      body into a new rcu_tasks_one_gp() function.  It also introduces
      a new ->tasks_gp_mutex to synchronize concurrent calls to this new
      rcu_tasks_one_gp() function.  This commit is preparation for allowing
      RCU tasks grace periods to be driven by the calling task during the
      mid-boot dead zone.
      
      Signed-off-by: default avatarPaul E. McKenney <paulmck@kernel.org>
      d96225fd
    • Paul E. McKenney's avatar
      rcu-tasks: Check for abandoned callbacks · 4cf0585c
      Paul E. McKenney authored
      
      
      This commit adds a debugging scan for callbacks that got lost during a
      callback-queueing transition.
      
      Signed-off-by: default avatarPaul E. McKenney <paulmck@kernel.org>
      4cf0585c
  3. Jun 20, 2022
  4. Jun 19, 2022