1. Jun 06, 2023
    • Peter Zijlstra's avatar
      arm64/arch_timer: Provide noinstr sched_clock_read() functions · 24ee7607
      Peter Zijlstra authored
      
      
      With the intent to provide local_clock_noinstr(), a variant of
      local_clock() that's safe to be called from noinstr code (with the
      assumption that any such code will already be non-preemptible),
      prepare for things by providing a noinstr sched_clock_read() function.
      
      Specifically, preempt_enable_*() calls out to schedule(), which upsets
      noinstr validation efforts.
      
      Signed-off-by: default avatarPeter Zijlstra (Intel) <peterz@infradead.org>
      Tested-by: Michael Kelley <mikelley@microsoft.com>  # Hyper-V
      Link: https://lore.kernel.org/r/20230519102715.435618812@infradead.org
      24ee7607
    • Peter Zijlstra's avatar
      arm64/io: Always inline all of __raw_{read,write}[bwlq]() · c1d26c0f
      Peter Zijlstra authored
      The next patch will want to use __raw_readl() from a noinstr section
      and as such that needs to be marked __always_inline to avoid the
      compiler being a silly bugger.
      
      Turns out it already is, but its siblings are not. Finish the work
      started in commit e43f1331
      
       ("arm64: Ask the compiler to
      __always_inline functions used by KVM at HYP") for consistenies sake.
      
      Signed-off-by: default avatarPeter Zijlstra (Intel) <peterz@infradead.org>
      Reviewed-by: default avatarValentin Schneider <vschneid@redhat.com>
      Tested-by: Michael Kelley <mikelley@microsoft.com>  # Hyper-V
      Link: https://lore.kernel.org/r/20230519102715.368919762@infradead.org
      c1d26c0f
    • Peter Zijlstra's avatar
      time/sched_clock: Provide sched_clock_noinstr() · 5949a68c
      Peter Zijlstra authored
      
      
      With the intent to provide local_clock_noinstr(), a variant of
      local_clock() that's safe to be called from noinstr code (with the
      assumption that any such code will already be non-preemptible),
      prepare for things by providing a noinstr sched_clock_noinstr() function.
      
      Specifically, preempt_enable_*() calls out to schedule(), which upsets
      noinstr validation efforts.
      
      As such, pull out the preempt_{dis,en}able_notrace() requirements from
      the sched_clock_read() implementations by explicitly providing it in
      the sched_clock() function.
      
      This further requires said sched_clock_read() functions to be noinstr
      themselves, for ARCH_WANTS_NO_INSTR users. See the next few patches.
      
      Signed-off-by: default avatarPeter Zijlstra (Intel) <peterz@infradead.org>
      Tested-by: Michael Kelley <mikelley@microsoft.com>  # Hyper-V
      Link: https://lore.kernel.org/r/20230519102715.302350330@infradead.org
      5949a68c
    • Peter Zijlstra's avatar
      seqlock/latch: Provide raw_read_seqcount_latch_retry() · d16317de
      Peter Zijlstra authored
      
      
      The read side of seqcount_latch consists of:
      
        do {
          seq = raw_read_seqcount_latch(&latch->seq);
          ...
        } while (read_seqcount_latch_retry(&latch->seq, seq));
      
      which is asymmetric in the raw_ department, and sure enough,
      read_seqcount_latch_retry() includes (explicit) instrumentation where
      raw_read_seqcount_latch() does not.
      
      This inconsistency becomes a problem when trying to use it from
      noinstr code. As such, fix it by renaming and re-implementing
      raw_read_seqcount_latch_retry() without the instrumentation.
      
      Specifically the instrumentation in question is kcsan_atomic_next(0)
      in do___read_seqcount_retry(). Loosing this annotation is not a
      problem because raw_read_seqcount_latch() does not pass through
      kcsan_atomic_next(KCSAN_SEQLOCK_REGION_MAX).
      
      Signed-off-by: default avatarPeter Zijlstra (Intel) <peterz@infradead.org>
      Reviewed-by: default avatarThomas Gleixner <tglx@linutronix.de>
      Reviewed-by: default avatarPetr Mladek <pmladek@suse.com>
      Tested-by: default avatarMichael Kelley <mikelley@microsof...>
      d16317de
    • Peter Zijlstra's avatar
      x86/sched: Rewrite topology setup · 8f2d6c41
      Peter Zijlstra authored
      
      
      Instead of having a number of fixed topologies to pick from; build one
      on the fly. This is both simpler now and simpler to extend in the
      future.
      
      Signed-off-by: default avatarPeter Zijlstra (Intel) <peterz@infradead.org>
      Link: https://lkml.kernel.org/r/20230601153522.GB559993%40hirez.programming.kicks-ass.net
      8f2d6c41
    • Peter Zijlstra's avatar
      sched: Consider task_struct::saved_state in wait_task_inactive() · 1c069187
      Peter Zijlstra authored
      With the introduction of task_struct::saved_state in commit
      5f220be2
      
       ("sched/wakeup: Prepare for RT sleeping spin/rwlocks")
      matching the task state has gotten more complicated. That same commit
      changed try_to_wake_up() to consider both states, but
      wait_task_inactive() has been neglected.
      
      Sebastian noted that the wait_task_inactive() usage in
      ptrace_check_attach() can misbehave when ptrace_stop() is blocked on
      the tasklist_lock after it sets TASK_TRACED.
      
      Therefore extract a common helper from ttwu_state_match() and use that
      to teach wait_task_inactive() about the PREEMPT_RT locks.
      
      Originally-by: default avatarSebastian Andrzej Siewior <bigeasy@linutronix.de>
      Signed-off-by: default avatarPeter Zijlstra (Intel) <peterz@infradead.org>
      Tested-by: default avatarSebastian Andrzej Siewior <bigeasy@linutronix.de>
      Link: https://lkml.kernel.org/r/20230601091234.GW83892@hirez.programming.kicks-ass.net
      1c069187
    • Peter Zijlstra's avatar
      sched: Unconditionally use full-fat wait_task_inactive() · d5e15866
      Peter Zijlstra authored
      
      
      While modifying wait_task_inactive() for PREEMPT_RT; the build robot
      noted that UP got broken. This led to audit and consideration of the
      UP implementation of wait_task_inactive().
      
      It looks like the UP implementation is also broken for PREEMPT;
      consider task_current_syscall() getting preempted between the two
      calls to wait_task_inactive().
      
      Therefore move the wait_task_inactive() implementation out of
      CONFIG_SMP and unconditionally use it.
      
      Signed-off-by: default avatarPeter Zijlstra (Intel) <peterz@infradead.org>
      Link: https://lkml.kernel.org/r/20230602103731.GA630648%40hirez.programming.kicks-ass.net
      d5e15866
    • Yicong Yang's avatar
      sched/fair: Don't balance task to its current running CPU · 0dd37d6d
      Yicong Yang authored
      
      
      We've run into the case that the balancer tries to balance a migration
      disabled task and trigger the warning in set_task_cpu() like below:
      
       ------------[ cut here ]------------
       WARNING: CPU: 7 PID: 0 at kernel/sched/core.c:3115 set_task_cpu+0x188/0x240
       Modules linked in: hclgevf xt_CHECKSUM ipt_REJECT nf_reject_ipv4 <...snip>
       CPU: 7 PID: 0 Comm: swapper/7 Kdump: loaded Tainted: G           O       6.1.0-rc4+ #1
       Hardware name: Huawei TaiShan 2280 V2/BC82AMDC, BIOS 2280-V2 CS V5.B221.01 12/09/2021
       pstate: 604000c9 (nZCv daIF +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
       pc : set_task_cpu+0x188/0x240
       lr : load_balance+0x5d0/0xc60
       sp : ffff80000803bc70
       x29: ffff80000803bc70 x28: ffff004089e190e8 x27: ffff004089e19040
       x26: ffff007effcabc38 x25: 0000000000000000 x24: 0000000000000001
       x23: ffff80000803be84 x22: 000000000000000c x21: ffffb093e79e2a78
       x20: 000000000000000c x19: ffff004089e19040 x18: 0000000000000000
       x17: 0000000000001fad x16: 0000000000000030 x15: 0000000000000000
       x14: 0000000000000003 x13: 0000000000000000 x12: 0000000000000000
       x11: 0000000000000001 x10: 0000000000000400 x9 : ffffb093e4cee530
       x8 : 00000000fffffffe x7 : 0000000000ce168a x6 : 000000000000013e
       x5 : 00000000ffffffe1 x4 : 0000000000000001 x3 : 0000000000000b2a
       x2 : 0000000000000b2a x1 : ffffb093e6d6c510 x0 : 0000000000000001
       Call trace:
        set_task_cpu+0x188/0x240
        load_balance+0x5d0/0xc60
        rebalance_domains+0x26c/0x380
        _nohz_idle_balance.isra.0+0x1e0/0x370
        run_rebalance_domains+0x6c/0x80
        __do_softirq+0x128/0x3d8
        ____do_softirq+0x18/0x24
        call_on_irq_stack+0x2c/0x38
        do_softirq_own_stack+0x24/0x3c
        __irq_exit_rcu+0xcc/0xf4
        irq_exit_rcu+0x18/0x24
        el1_interrupt+0x4c/0xe4
        el1h_64_irq_handler+0x18/0x2c
        el1h_64_irq+0x74/0x78
        arch_cpu_idle+0x18/0x4c
        default_idle_call+0x58/0x194
        do_idle+0x244/0x2b0
        cpu_startup_entry+0x30/0x3c
        secondary_start_kernel+0x14c/0x190
        __secondary_switched+0xb0/0xb4
       ---[ end trace 0000000000000000 ]---
      
      Further investigation shows that the warning is superfluous, the migration
      disabled task is just going to be migrated to its current running CPU.
      This is because that on load balance if the dst_cpu is not allowed by the
      task, we'll re-select a new_dst_cpu as a candidate. If no task can be
      balanced to dst_cpu we'll try to balance the task to the new_dst_cpu
      instead. In this case when the migration disabled task is not on CPU it
      only allows to run on its current CPU, load balance will select its
      current CPU as new_dst_cpu and later triggers the warning above.
      
      The new_dst_cpu is chosen from the env->dst_grpmask. Currently it
      contains CPUs in sched_group_span() and if we have overlapped groups it's
      possible to run into this case. This patch makes env->dst_grpmask of
      group_balance_mask() which exclude any CPUs from the busiest group and
      solve the issue. For balancing in a domain with no overlapped groups
      the behaviour keeps same as before.
      
      Suggested-by: default avatarVincent Guittot <vincent.guittot@linaro.org>
      Signed-off-by: default avatarYicong Yang <yangyicong@hisilicon.com>
      Signed-off-by: default avatarPeter Zijlstra (Intel) <peterz@infradead.org>
      Reviewed-by: default avatarVincent Guittot <vincent.guittot@linaro.org>
      Link: https://lore.kernel.org/r/20230530082507.10444-1-yangyicong@huawei.com
      0dd37d6d
  2. May 31, 2023
  3. May 20, 2023
  4. May 08, 2023
    • 晏艳(采苓)'s avatar
      sched/debug: Correct printing for rq->nr_uninterruptible · a6fcdd8d
      晏艳(采苓) authored
      Commit e6fe3f42
      
       ("sched: Make multiple runqueue task counters
      32-bit") changed the type for rq->nr_uninterruptible from "unsigned
      long" to "unsigned int", but left wrong cast print to
      /sys/kernel/debug/sched/debug and to the console.
      
      For example, nr_uninterruptible's value is fffffff7 with type
      "unsigned int", (long)nr_uninterruptible shows 4294967287 while
      (int)nr_uninterruptible prints -9. So using int cast fixes wrong
      printing.
      
      Signed-off-by: default avatarYan Yan <yanyan.yan@antgroup.com>
      Signed-off-by: default avatarPeter Zijlstra (Intel) <peterz@infradead.org>
      Link: https://lkml.kernel.org/r/20230506074253.44526-1-yanyan.yan@antgroup.com
      a6fcdd8d
    • Tim C Chen's avatar
      sched/topology: Propagate SMT flags when removing degenerate domain · bf2dc42d
      Tim C Chen authored
      
      
      When a degenerate cluster domain for core with SMT CPUs is removed,
      the SD_SHARE_CPUCAPACITY flag in the local child sched group was not
      propagated to the new parent.  We need this flag to properly determine
      whether the local sched group is SMT.  Set the flag in the local
      child sched group of the new parent sched domain.
      
      Signed-off-by: default avatarTim Chen <tim.c.chen@linux.intel.com>
      Signed-off-by: default avatarPeter Zijlstra (Intel) <peterz@infradead.org>
      Reviewed-by: default avatarRicardo Neri <ricardo.neri-calderon@linux.intel.com>
      Link: https://lkml.kernel.org/r/73cf0959eafa53c02e7ef6bf805d751d9190e55d.1683156492.git.tim.c.chen@linux.intel.com
      bf2dc42d
    • Suren Baghdasaryan's avatar
      psi: remove 500ms min window size limitation for triggers · 519fabc7
      Suren Baghdasaryan authored
      
      
      Current 500ms min window size for psi triggers limits polling interval
      to 50ms to prevent polling threads from using too much cpu bandwidth by
      polling too frequently. However the number of cgroups with triggers is
      unlimited, so this protection can be defeated by creating multiple
      cgroups with psi triggers (triggers in each cgroup are served by a single
      "psimon" kernel thread).
      Instead of limiting min polling period, which also limits the latency of
      psi events, it's better to limit psi trigger creation to authorized users
      only, like we do for system-wide psi triggers (/proc/pressure/* files can
      be written only by processes with CAP_SYS_RESOURCE capability). This also
      makes access rules for cgroup psi files consistent with system-wide ones.
      Add a CAP_SYS_RESOURCE capability check for cgroup psi file writers and
      remove the psi window min size limitation.
      
      Suggested-by: default avatarSudarshan Rajagopalan <quic_sudaraja@quicinc.com>
      Signed-off-by: default avatarSuren Baghdasaryan <surenb@google.com>
      Signed-off-by: default avatarPeter Zijlstra (Intel) <peterz@infradead.org>
      Acked-by: default avatarMichal Hocko <mhocko@suse.com>
      Acked-by: default avatarJohannes Weiner <hannes@cmpxchg.org>
      Link: https://lore.kernel.org/all/cover.1676067791.git.quic_sudaraja@quicinc.com/
      519fabc7
    • Chen Yu's avatar
      x86/sched: Add the SD_ASYM_PACKING flag to the die domain of hybrid processors · 044f0e27
      Chen Yu authored
      
      
      Intel Meteor Lake hybrid processors have cores in two separate dies. The
      cores in one of the dies have higher maximum frequency. Use the SD_ASYM_
      PACKING flag to give higher priority to the die with CPUs of higher maximum
      frequency.
      
      Suggested-by: default avatarRicardo Neri <ricardo.neri-calderon@linux.intel.com>
      Signed-off-by: default avatarChen Yu <yu.c.chen@intel.com>
      Signed-off-by: default avatarPeter Zijlstra (Intel) <peterz@infradead.org>
      Link: https://lore.kernel.org/r/20230406203148.19182-13-ricardo.neri-calderon@linux.intel.com
      044f0e27
    • Ricardo Neri's avatar
      x86/sched/itmt: Give all SMT siblings of a core the same priority · 046a5a95
      Ricardo Neri authored
      
      
      X86 does not have the SD_ASYM_PACKING flag in the SMT domain. The scheduler
      knows how to handle SMT and non-SMT cores of different priority. There is
      no reason for SMT siblings of a core to have different priorities.
      
      Signed-off-by: default avatarRicardo Neri <ricardo.neri-calderon@linux.intel.com>
      Signed-off-by: default avatarPeter Zijlstra (Intel) <peterz@infradead.org>
      Reviewed-by: default avatarLen Brown <len.brown@intel.com>
      Tested-by: default avatarZhang Rui <rui.zhang@intel.com>
      Link: https://lore.kernel.org/r/20230406203148.19182-12-ricardo.neri-calderon@linux.intel.com
      046a5a95
    • Ricardo Neri's avatar
      x86/sched: Remove SD_ASYM_PACKING from the SMT domain flags · 995998eb
      Ricardo Neri authored
      
      
      There is no difference between any of the SMT siblings of a physical core.
      Do not do asym_packing load balancing at this level.
      
      Signed-off-by: default avatarRicardo Neri <ricardo.neri-calderon@linux.intel.com>
      Signed-off-by: default avatarPeter Zijlstra (Intel) <peterz@infradead.org>
      Tested-by: default avatarZhang Rui <rui.zhang@intel.com>
      Link: https://lore.kernel.org/r/20230406203148.19182-11-ricardo.neri-calderon@linux.intel.com
      995998eb
    • Ricardo Neri's avatar
      sched/topology: Remove SHARED_CHILD from ASYM_PACKING · ca528cc5
      Ricardo Neri authored
      
      
      Only x86 and Power7 use ASYM_PACKING. They use it differently.
      
      Power7 has cores of equal priority, but the SMT siblings of a core have
      different priorities. Parent scheduling domains do not need (nor have) the
      ASYM_PACKING flag. SHARED_CHILD is not needed. Using SHARED_PARENT would
      cause the topology debug code to complain.
      
      X86 has cores of different priority, but all the SMT siblings of the core
      have equal priority. It needs ASYM_PACKING at the MC level, but not at the
      SMT level (it also needs it at upper levels if they have scheduling groups
      of different priority). Removing ASYM_PACKING from the SMT domain causes
      the topology debug code to complain.
      
      Remove SHARED_CHILD for now. We still need a topology check that satisfies
      both architectures.
      
      Suggested-by: default avatarValentin Schneider <vschneid@redhat.com>
      Signed-off-by: default avatarRicardo Neri <ricardo.neri-calderon@linux.intel.com>
      Signed-off-by: default avatarPeter Zijlstra (Intel) <peterz@infradead.org>
      Tested-by: default avatarZhang Rui <rui.zhang@intel.com>
      Link: https://lore.kernel.org/r/20230406203148.19182-10-ricardo.neri-calderon@linux.intel.com
      ca528cc5
    • Ricardo Neri's avatar
      sched/topology: Check SDF_SHARED_CHILD in highest_flag_domain() · 40b4d3dc
      Ricardo Neri authored
      
      
      Do not assume that all the children of a scheduling domain have a given
      flag. Check whether it has the SDF_SHARED_CHILD meta flag.
      
      Suggested-by: default avatarIonela Voinescu <ionela.voinescu@arm.com>
      Signed-off-by: default avatarRicardo Neri <ricardo.neri-calderon@linux.intel.com>
      Signed-off-by: default avatarPeter Zijlstra (Intel) <peterz@infradead.org>
      Link: https://lore.kernel.org/r/20230406203148.19182-9-ricardo.neri-calderon@linux.intel.com
      40b4d3dc
    • Ricardo Neri's avatar
      sched/fair: Do not even the number of busy CPUs via asym_packing · c9ca0788
      Ricardo Neri authored
      
      
      Now that find_busiest_group() triggers load balancing between a fully_
      busy SMT2 core and an idle non-SMT core, it is no longer needed to force
      balancing via asym_packing. Use asym_packing only as intended: when there
      is high-priority CPU that is idle.
      
      After this change, the same logic apply to SMT and non-SMT local groups.
      It makes less sense having a separate function to deal specifically with
      SMT. Fold the logic in asym_smt_can_pull_tasks() into sched_asym().
      
      Signed-off-by: default avatarRicardo Neri <ricardo.neri-calderon@linux.intel.com>
      Signed-off-by: default avatarPeter Zijlstra (Intel) <peterz@infradead.org>
      Tested-by: default avatarZhang Rui <rui.zhang@intel.com>
      Link: https://lore.kernel.org/r/20230406203148.19182-8-ricardo.neri-calderon@linux.intel.com
      c9ca0788
    • Ricardo Neri's avatar
      sched/fair: Use the busiest group to set prefer_sibling · 43726bde
      Ricardo Neri authored
      
      
      The prefer_sibling setting acts on the busiest group to move excess tasks
      to the local group. This should be done as per request of the child of the
      busiest group's sched domain, not the local group's.
      
      Using the flags of the child domain of the local group works fortuitously
      if both groups have child domains.
      
      There are cases, however, in which the busiest group's sched domain has
      child but the local group's does not. Consider, for instance a non-SMT
      core (or an SMT core with only one online sibling) doing load balance with
      an SMT core at the MC level. SD_PREFER_SIBLING of the busiest group's child
      domain will not be honored. We are left with a fully busy SMT core and an
      idle non-SMT core.
      
      Suggested-by: default avatarDietmar Eggemann <dietmar.eggemann@arm.com>
      Signed-off-by: default avatarRicardo Neri <ricardo.neri-calderon@linux.intel.com>
      Signed-off-by: default avatarPeter Zijlstra (Intel) <peterz@infradead.org>
      Tested-by: default avatarZhang Rui <rui.zhang@intel.com>
      Link: https://lore.kernel.org/r/20230406203148.19182-7-ricardo.neri-calderon@linux.intel.com
      43726bde
    • Ricardo Neri's avatar
      sched/fair: Keep a fully_busy SMT sched group as busiest · 5fd6d7f4
      Ricardo Neri authored
      
      
      When comparing two fully_busy scheduling groups, keep the current busiest
      group if it represents an SMT core. Tasks in such scheduling group share
      CPU resources and need more help than tasks in a non-SMT fully_busy group.
      
      Signed-off-by: default avatarRicardo Neri <ricardo.neri-calderon@linux.intel.com>
      Signed-off-by: default avatarPeter Zijlstra (Intel) <peterz@infradead.org>
      Tested-by: default avatarZhang Rui <rui.zhang@intel.com>
      Link: https://lore.kernel.org/r/20230406203148.19182-6-ricardo.neri-calderon@linux.intel.com
      5fd6d7f4
    • Ricardo Neri's avatar
      sched/fair: Let low-priority cores help high-priority busy SMT cores · 18ad3453
      Ricardo Neri authored
      
      
      Using asym_packing priorities within an SMT core is straightforward. Just
      follow the priorities that hardware indicates.
      
      When balancing load from an SMT core, also consider the idle state of its
      siblings. Priorities do not reflect that an SMT core divides its throughput
      among all its busy siblings. They only makes sense when exactly one sibling
      is busy.
      
      Indicate that active balance is needed if the destination CPU has lower
      priority than the source CPU but the latter has busy SMT siblings.
      
      Make find_busiest_queue() not skip higher-priority SMT cores with more than
      busy sibling.
      
      Suggested-by: default avatarValentin Schneider <vschneid@redhat.com>
      Signed-off-by: default avatarRicardo Neri <ricardo.neri-calderon@linux.intel.com>
      Signed-off-by: default avatarPeter Zijlstra (Intel) <peterz@infradead.org>
      Tested-by: default avatarZhang Rui <rui.zhang@intel.com>
      Link: https://lore.kernel.org/r/20230406203148.19182-5-ricardo.neri-calderon@linux.intel.com
      18ad3453
    • Ricardo Neri's avatar
      sched/fair: Simplify asym_packing logic for SMT cores · ef7657d4
      Ricardo Neri authored
      
      
      Callers of asym_smt_can_pull_tasks() check the idle state of the
      destination CPU and its SMT siblings, if any. No extra checks are needed
      in such function.
      
      Since SMT cores divide capacity among its siblings, priorities only really
      make sense if only one sibling is active. This is true for SMT2, SMT4,
      SMT8, etc. Do not use asym_packing load balance for this case. Instead,
      let find_busiest_group() handle imbalances.
      
      When balancing non-SMT cores or at higher scheduling domains (e.g.,
      between MC scheduling groups), continue using priorities.
      
      Signed-off-by: default avatarRicardo Neri <ricardo.neri-calderon@linux.intel.com>
      Signed-off-by: default avatarPeter Zijlstra (Intel) <peterz@infradead.org>
      Reviewed-by: default avatarLen Brown <len.brown@intel.com>
      Tested-by: default avatarZhang Rui <rui.zhang@intel.com>
      Link: https://lore.kernel.org/r/20230406203148.19182-4-ricardo.neri-calderon@linux.intel.com
      ef7657d4
    • Ricardo Neri's avatar
      sched/fair: Only do asym_packing load balancing from fully idle SMT cores · eefefa71
      Ricardo Neri authored
      
      
      When balancing load between cores, all the SMT siblings of the destination
      CPU, if any, must be idle. Otherwise, pulling new tasks degrades the
      throughput of the busy SMT siblings. The overall throughput of the system
      remains the same.
      
      When balancing load within an SMT core this consideration is not relevant.
      Follow the priorities that hardware indicates.
      
      Suggested-by: default avatarValentin Schneider <vschneid@redhat.com>
      Signed-off-by: default avatarRicardo Neri <ricardo.neri-calderon@linux.intel.com>
      Signed-off-by: default avatarPeter Zijlstra (Intel) <peterz@infradead.org>
      Tested-by: default avatarZhang Rui <rui.zhang@intel.com>
      Link: https://lore.kernel.org/r/20230406203148.19182-3-ricardo.neri-calderon@linux.intel.com
      eefefa71
    • Ricardo Neri's avatar
      sched/fair: Move is_core_idle() out of CONFIG_NUMA · 8b36d07f
      Ricardo Neri authored
      
      
      asym_packing needs this function to determine whether an SMT core is a
      suitable destination for load balancing.
      
      Signed-off-by: default avatarRicardo Neri <ricardo.neri-calderon@linux.intel.com>
      Signed-off-by: default avatarPeter Zijlstra (Intel) <peterz@infradead.org>
      Tested-by: default avatarZhang Rui <rui.zhang@intel.com>
      Link: https://lore.kernel.org/r/20230406203148.19182-2-ricardo.neri-calderon@linux.intel.com
      8b36d07f
    • Linus Torvalds's avatar
      Linux 6.4-rc1 · ac9a7868
      Linus Torvalds authored
      ac9a7868
    • Linus Torvalds's avatar
      Merge tag 'perf-tools-for-v6.4-3-2023-05-06' of... · f085df1b
      Linus Torvalds authored
      Merge tag 'perf-tools-for-v6.4-3-2023-05-06' of git://git.kernel.org/pub/scm/linux/kernel/git/acme/linux
      
      Pull perf tool updates from Arnaldo Carvalho de Melo:
       "Third version of perf tool updates, with the build problems with with
        using a 'vmlinux.h' generated from the main build fixed, and the bpf
        skeleton build disabled by default.
      
        Build:
      
         - Require libtraceevent to build, one can disable it using
           NO_LIBTRACEEVENT=1.
      
           It is required for tools like 'perf sched', 'perf kvm', 'perf
           trace', etc.
      
           libtraceevent is available in most distros so installing
           'libtraceevent-devel' should be a one-time event to continue
           building perf as usual.
      
           Using NO_LIBTRACEEVENT=1 produces tooling that is functional and
           sufficient for lots of users not interested in those libtraceevent
           dependent features.
      
         - Allow Python support in 'perf script' when libtraceevent isn't
           linked, as not all features requires it, for instance Intel PT does
           not use tracepoints.
      
         - Error if the python interpreter needed for jevents to work isn't
           available and NO_JEVENTS=1 isn't set, preventing a build without
           support for JSON vendor events, which is a rare but possible
           condition. The two check error messages:
      
              $(error ERROR: No python interpreter needed for jevents generation. Install python or build with NO_JEVENTS=1.)
              $(error ERROR: Python interpreter needed for jevents generation too old (older than 3.6). Install a newer python or build with NO_JEVENTS=1.)
      
         - Make libbpf 1.0 the minimum required when building with out of
           tree, distro provided libbpf.
      
         - Use libsdtc++'s and LLVM's libcxx's __cxa_demangle, a portable C++
           demangler, add 'perf test' entry for it.
      
         - Make binutils libraries opt in, as distros disable building with it
           due to licensing, they were used for C++ demangling, for instance.
      
         - Switch libpfm4 to opt-out rather than opt-in, if libpfm-devel (or
           equivalent) isn't installed, we'll just have a build warning:
      
             Makefile.config:1144: libpfm4 not found, disables libpfm4 support. Please install libpfm4-dev
      
         - Add a feature test for scandirat(), that is not implemented so far
           in musl and uclibc, disabling features that need it, such as
           scanning for tracepoints in /sys/kernel/tracing/events.
      
        perf BPF filters:
      
         - New feature where BPF can be used to filter samples, for instance:
      
            $ sudo ./perf record -e cycles --filter 'period > 1000' true
            $ sudo ./perf script
                 perf-exec 2273949 546850.708501:       5029 cycles:  ffffffff826f9e25 finish_wait+0x5 ([kernel.kallsyms])
                 perf-exec 2273949 546850.708508:      32409 cycles:  ffffffff826f9e25 finish_wait+0x5 ([kernel.kallsyms])
                 perf-exec 2273949 546850.708526:     143369 cycles:  ffffffff82b4cdbf xas_start+0x5f ([kernel.kallsyms])
                 perf-exec 2273949 546850.708600:     372650 cycles:  ffffffff8286b8f7 __pagevec_lru_add+0x117 ([kernel.kallsyms])
                 perf-exec 2273949 546850.708791:     482953 cycles:  ffffffff829190de __mod_memcg_lruvec_state+0x4e ([kernel.kallsyms])
                      true 2273949 546850.709036:     501985 cycles:  ffffffff828add7c tlb_gather_mmu+0x4c ([kernel.kallsyms])
                      true 2273949 546850.709292:     503065 cycles:      7f2446d97c03 _dl_map_object_deps+0x973 (/usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2)
      
         - In addition to 'period' (PERF_SAMPLE_PERIOD), the other
           PERF_SAMPLE_ can be used for filtering, and also some other sample
           accessible values, from tools/perf/Documentation/perf-record.txt:
      
              Essentially the BPF filter expression is:
      
              <term> <operator> <value> (("," | "||") <term> <operator> <value>)*
      
           The <term> can be one of:
              ip, id, tid, pid, cpu, time, addr, period, txn, weight, phys_addr,
              code_pgsz, data_pgsz, weight1, weight2, weight3, ins_lat, retire_lat,
              p_stage_cyc, mem_op, mem_lvl, mem_snoop, mem_remote, mem_lock,
              mem_dtlb, mem_blk, mem_hops
      
           The <operator> can be one of:
              ==, !=, >, >=, <, <=, &
      
           The <value> can be one of:
              <number> (for any term)
              na, load, store, pfetch, exec (for mem_op)
              l1, l2, l3, l4, cxl, io, any_cache, lfb, ram, pmem (for mem_lvl)
              na, none, hit, miss, hitm, fwd, peer (for mem_snoop)
              remote (for mem_remote)
              na, locked (for mem_locked)
              na, l1_hit, l1_miss, l2_hit, l2_miss, any_hit, any_miss, walk, fault (for mem_dtlb)
              na, by_data, by_addr (for mem_blk)
              hops0, hops1, hops2, hops3 (for mem_hops)
      
        perf lock contention:
      
         - Show lock type with address.
      
         - Track and show mmap_lock, siglock and per-cpu rq_lock with address.
           This is done for mmap_lock by following the current->mm pointer:
      
            $ sudo ./perf lock con -abl -- sleep 10
             contended   total wait     max wait     avg wait            address   symbol
             ...
                 16344    312.30 ms      2.22 ms     19.11 us   ffff8cc702595640
                 17686    310.08 ms      1.49 ms     17.53 us   ffff8cc7025952c0
                     3     84.14 ms     45.79 ms     28.05 ms   ffff8cc78114c478   mmap_lock
                  3557     76.80 ms     68.75 us     21.59 us   ffff8cc77ca3af58
                     1     68.27 ms     68.27 ms     68.27 ms   ffff8cda745dfd70
                     9     54.53 ms      7.96 ms      6.06 ms   ffff8cc7642a48b8   mmap_lock
                 14629     44.01 ms     60.00 us      3.01 us   ffff8cc7625f9ca0
                  3481     42.63 ms    140.71 us     12.24 us   ffffffff937906ac   vmap_area_lock
                 16194     38.73 ms     42.15 us      2.39 us   ffff8cd397cbc560
                    11     38.44 ms     10.39 ms      3.49 ms   ffff8ccd6d12fbb8   mmap_lock
                     1      5.43 ms      5.43 ms      5.43 ms   ffff8cd70018f0d8
                  1674      5.38 ms    422.93 us      3.21 us   ffffffff92e06080   tasklist_lock
                   581      4.51 ms    130.68 us      7.75 us   ffff8cc9b1259058
                     5      3.52 ms      1.27 ms    703.23 us   ffff8cc754510070
                   112      3.47 ms     56.47 us     31.02 us   ffff8ccee38b3120
                   381      3.31 ms     73.44 us      8.69 us   ffffffff93790690   purge_vmap_area_lock
                   255      3.19 ms     36.35 us     12.49 us   ffff8d053ce30c80
      
         - Update default map size to 16384.
      
         - Allocate single letter option -M for --map-nr-entries, as it is
           proving being frequently used.
      
         - Fix struct rq lock access for older kernels with BPF's CO-RE
           (Compile once, run everywhere).
      
         - Fix problems found with MSAn.
      
        perf report/top:
      
         - Add inline information when using --call-graph=fp or lbr, as was
           already done to the --call-graph=dwarf callchain mode.
      
         - Improve the 'srcfile' sort key performance by really using an
           optimization introduced in 6.2 for the 'srcline' sort key that
           avoids calling addr2line for comparision with each sample.
      
        perf sched:
      
         - Make 'perf sched latency/map/replay' to use "sched:sched_waking"
           instead of "sched:sched_waking", consistent with 'perf record'
           since d566a9c2 ("perf sched: Prefer sched_waking event when it
           exists").
      
        perf ftrace:
      
         - Make system wide the default target for latency subcommand, run the
           following command then generate some network traffic and press
           control+C:
      
             # perf ftrace latency -T __kfree_skb
           ^C
               DURATION     |      COUNT | GRAPH                                          |
                0 - 1    us |         27 | #############                                  |
                1 - 2    us |         22 | ###########                                    |
                2 - 4    us |          8 | ####                                           |
                4 - 8    us |          5 | ##                                             |
                8 - 16   us |         24 | ############                                   |
               16 - 32   us |          2 | #                                              |
               32 - 64   us |          1 |                                                |
               64 - 128  us |          0 |                                                |
              128 - 256  us |          0 |                                                |
              256 - 512  us |          0 |                                                |
              512 - 1024 us |          0 |                                                |
                1 - 2    ms |          0 |                                                |
                2 - 4    ms |          0 |                                                |
                4 - 8    ms |          0 |                                                |
                8 - 16   ms |          0 |                                                |
               16 - 32   ms |          0 |                                                |
               32 - 64   ms |          0 |                                                |
               64 - 128  ms |          0 |                                                |
              128 - 256  ms |          0 |                                                |
              256 - 512  ms |          0 |                                                |
              512 - 1024 ms |          0 |                                                |
                1 - ...   s |          0 |                                                |
             #
      
        perf top:
      
         - Add --branch-history (LBR: Last Branch Record) option, just like
           already available for 'perf record'.
      
         - Fix segfault in thread__comm_len() where thread->comm was being
           used outside thread->comm_lock.
      
        perf annotate:
      
         - Allow configuring objdump and addr2line in ~/.perfconfig., so that
           you can use alternative binaries, such as llvm's.
      
        perf kvm:
      
         - Add TUI mode for 'perf kvm stat report'.
      
        Reference counting:
      
         - Add reference count checking infrastructure to check for use after
           free, done to the 'cpumap', 'namespaces', 'maps' and 'map' structs,
           more to come.
      
           To build with it use -DREFCNT_CHECKING=1 in the make command line
           to build tools/perf. Documented at:
      
             https://perf.wiki.kernel.org/index.php/Reference_Count_Checking
      
         - The above caught, for instance, fix, present in this series:
      
              - Fix maps use after put in 'perf test "Share thread maps"':
      
                'maps' is copied from leader, but the leader is put on line 79
                and then 'maps' is used to read the reference count below - so
                a use after put, with the put of maps happening within
                thread__put.
      
           Fixed by reversing the order of puts so that the leader is put
           last.
      
         - Also several fixes were made to places where reference counts were
           not being held.
      
         - Make this one of the tests in 'make -C tools/perf build-test' to
           regularly build test it and to make sure no direct access to the
           reference counted structs are made, doing that via accessors to
           check the validity of the struct pointer.
      
        ARM64:
      
         - Fix 'perf report' segfault when filtering coresight traces by
           sparse lists of CPUs.
      
         - Add support for 'simd' as a sort field for 'perf report', to show
           ARM's NEON SIMD's predicate flags: "partial" and "empty".
      
        arm64 vendor events:
      
         - Add N1 metrics.
      
        Intel vendor events:
      
         - Add graniterapids, grandridge and sierraforrest events.
      
         - Refresh events for: alderlake, aldernaken, broadwell, broadwellde,
           broadwellx, cascadelakx, haswell, haswellx, icelake, icelakex,
           jaketown, meteorlake, knightslanding, sandybridge, sapphirerapids,
           silvermont, skylake, tigerlake and westmereep-dp
      
         - Refresh metrics for alderlake-n, broadwell, broadwellde,
           broadwellx, haswell, haswellx, icelakex, ivybridge, ivytown and
           skylakex.
      
        perf stat:
      
         - Implement --topdown using JSON metrics.
      
         - Add TopdownL1 JSON metric as a default if present, but disable it
           for now for some Intel hybrid architectures, a series of patches
           addressing this is being reviewed and will be submitted for v6.5.
      
         - Use metrics for --smi-cost.
      
         - Update topdown documentation.
      
        Vendor events (JSON) infrastructure:
      
         - Add support for computing and printing metric threshold values. For
           instance, here is one found in thesapphirerapids json file:
      
             {
                 "BriefDescription": "Percentage of cycles spent in System Management Interrupts.",
                 "MetricExpr": "((msr@aperf@ - cycles) / msr@aperf@ if msr@smi@ > 0 else 0)",
                 "MetricGroup": "smi",
                 "MetricName": "smi_cycles",
                 "MetricThreshold": "smi_cycles > 0.1",
                 "ScaleUnit": "100%"
             },
      
         - Test parsing metric thresholds with the fake PMU in 'perf test
           pmu-events'.
      
         - Support for printing metric thresholds in 'perf list'.
      
         - Add --metric-no-threshold option to 'perf stat'.
      
         - Add rand (reverse and) and has_pmem (optane memory) support to
           metrics.
      
         - Sort list of input files to avoid depending on the order from
           readdir() helping in obtaining reproducible builds.
      
        S/390:
      
         - Add common metrics: - CPI (cycles per instruction), prbstate (ratio
           of instructions executed in problem state compared to total number
           of instructions), l1mp (Level one instruction and data cache misses
           per 100 instructions).
      
         - Add cache metrics for z13, z14, z15 and z16.
      
         - Add metric for TLB and cache.
      
        ARM:
      
         - Add raw decoding for SPE (Statistical Profiling Extension) v1.3 MTE
           (Memory Tagging Extension) and MOPS (Memory Operations) load/store.
      
        Intel PT hardware tracing:
      
         - Add event type names UINTR (User interrupt delivered) and UIRET
           (Exiting from user interrupt routine), documented in table 32-50
           "CFE Packet Type and Vector Fields Details" in the Intel Processor
           Trace chapter of The Intel SDM Volume 3 version 078.
      
         - Add support for new branch instructions ERETS and ERETU.
      
         - Fix CYC timestamps after standalone CBR
      
        ARM CoreSight hardware tracing:
      
         - Allow user to override timestamp and contextid settings.
      
         - Fix segfault in dso lookup.
      
         - Fix timeless decode mode detection.
      
         - Add separate decode paths for timeless and per-thread modes.
      
        auxtrace:
      
         - Fix address filter entire kernel size.
      
        Miscellaneous:
      
         - Fix use-after-free and unaligned bugs in the PLT handling routines.
      
         - Use zfree() to reduce chances of use after free.
      
         - Add missing 0x prefix for addresses printed in hexadecimal in 'perf
           probe'.
      
         - Suppress massive unsupported target platform errors in the unwind
           code.
      
         - Fix return incorrect build_id size in elf_read_build_id().
      
         - Fix 'perf scripts intel-pt-events.py' IPC output for Python 2 .
      
         - Add missing new parameter in kfree_skb tracepoint to the python
           scripts using it.
      
         - Add 'perf bench syscall fork' benchmark.
      
         - Add support for printing PERF_MEM_LVLNUM_UNC (Uncached access) in
           'perf mem'.
      
         - Fix wrong size expectation for perf test 'Setup struct
           perf_event_attr' caused by the patch adding
           perf_event_attr::config3.
      
         - Fix some spelling mistakes"
      
      * tag 'perf-tools-for-v6.4-3-2023-05-06' of git://git.kernel.org/pub/scm/linux/kernel/git/acme/linux: (365 commits)
        Revert "perf build: Make BUILD_BPF_SKEL default, rename to NO_BPF_SKEL"
        Revert "perf build: Warn for BPF skeletons if endian mismatches"
        perf metrics: Fix SEGV with --for-each-cgroup
        perf bpf skels: Stop using vmlinux.h generated from BTF, use subset of used structs + CO-RE
        perf stat: Separate bperf from bpf_profiler
        perf test record+probe_libc_inet_pton: Fix call chain match on x86_64
        perf test record+probe_libc_inet_pton: Fix call chain match on s390
        perf tracepoint: Fix memory leak in is_valid_tracepoint()
        perf cs-etm: Add fix for coresight trace for any range of CPUs
        perf build: Fix unescaped # in perf build-test
        perf unwind: Suppress massive unsupported target platform errors
        perf script: Add new parameter in kfree_skb tracepoint to the python scripts using it
        perf script: Print raw ip instead of binary offset for callchain
        perf symbols: Fix return incorrect build_id size in elf_read_build_id()
        perf list: Modify the warning message about scandirat(3)
        perf list: Fix memory leaks in print_tracepoint_events()
        perf lock contention: Rework offset calculation with BPF CO-RE
        perf lock contention: Fix struct rq lock access
        perf stat: Disable TopdownL1 on hybrid
        perf stat: Avoid SEGV on counter->name
        ...
      f085df1b
    • Linus Torvalds's avatar
      Merge tag 'core-debugobjects-2023-05-06' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip · 17784de6
      Linus Torvalds authored
      Pull debugobjects fix from Thomas Gleixner:
       "A single fix for debugobjects:
      
        The recent fix to ensure atomicity of lookup and allocation
        inadvertently broke the pool refill mechanism, so that debugobject
        OOMs now in certain situations. The reason is that the functions which
        got updated no longer invoke debug_objecs_init(), which is now the
        only place to care about refilling the tracking object pool.
      
        Restore the original behaviour by adding explicit refill opportunities
        to those places"
      
      * tag 'core-debugobjects-2023-05-06' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
        debugobject: Ensure pool refill (again)
      17784de6
    • Linus Torvalds's avatar
      Merge tag 'v6.4-p2' of git://git.kernel.org/pub/scm/linux/kernel/git/herbert/crypto-2.6 · 6f69c981
      Linus Torvalds authored
      Pull crypto fixes from Herbert Xu:
      
       - A long-standing bug in crypto_engine
      
       - A buggy but harmless check in the sun8i-ss driver
      
       - A regression in the CRYPTO_USER interface
      
      * tag 'v6.4-p2' of git://git.kernel.org/pub/scm/linux/kernel/git/herbert/crypto-2.6:
        crypto: api - Fix CRYPTO_USER checks for report function
        crypto: engine - fix crypto_queue backlog handling
        crypto: sun8i-ss - Fix a test in sun8i_ss_setup_ivs()
      6f69c981
    • Linus Torvalds's avatar
      Merge tag '6.4-rc-smb3-client-fixes-part2' of git://git.samba.org/sfrench/cifs-2.6 · 63342b1d
      Linus Torvalds authored
      Pull cifs fixes from Steve French:
       "smb3 client fixes, mostly DFS or reconnect related:
      
         - Two DFS connection sharing fixes
      
         - DFS refresh fix
      
         - Reconnect fix
      
         - Two potential use after free fixes
      
         - Also print prefix patch in mount debug msg
      
         - Two small cleanup fixes"
      
      * tag '6.4-rc-smb3-client-fixes-part2' of git://git.samba.org/sfrench/cifs-2.6:
        cifs: Remove unneeded semicolon
        cifs: fix sharing of DFS connections
        cifs: avoid potential races when handling multiple dfs tcons
        cifs: protect access of TCP_Server_Info::{origin,leaf}_fullpath
        cifs: fix potential race when tree connecting ipc
        cifs: fix potential use-after-free bugs in TCP_Server_Info::hostname
        cifs: print smb3_fs_context::source when mounting
        cifs: protect session status check in smb2_reconnect()
        SMB3.1.1: correct definition for app_instance_id create contexts
      63342b1d
    • Linus Torvalds's avatar
      Merge tag 'clk-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/clk/linux · d6b8a8c4
      Linus Torvalds authored
      Pull clk fixes from Stephen Boyd:
       "A couple more patches that would be good to get into -rc1:
      
         - Revert an i.MX patch that's causing video failures because division
           math goes sideways
      
         - Fix a clang + W=1 build isue where FIELD_PREP() is taking a 32-bit
           variable instead of the usual u64 type
      
         - Fix a Kconfig bug in the StarFive JH7110 clk config that selects a
           reset controller when it can't be selected"
      
      * tag 'clk-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/clk/linux:
        clk: starfive: Fix RESET_STARFIVE_JH7110 can't be selected in a specified case
        clk: sp7021: Adjust width of _m in HWM_FIELD_PREP()
        Revert "clk: imx: composite-8m: Add support to determine_rate"
      d6b8a8c4
    • Linus Torvalds's avatar
      Merge tag 'mailbox-v6.4' of git://git.linaro.org/landing-teams/working/fujitsu/integration · 1c1094e4
      Linus Torvalds authored
      Pull mailbox updates from Jassi Brar:
      
       - mailbox api: allow direct registration to a channel and convert omap
         and pcc to use mbox_bind_client
      
       - omap and hi6220 : use of_property_read_bool
      
       - test: fix double-free and use spinlock header
      
       - rockchip and bcm-pdc: drop of_match_ptr
      
       - mpfs: change config symbol
      
       - mediatek gce: support MT6795
      
       - qcom apcs: consolidate of_device_id and support IPQ9574
      
      * tag 'mailbox-v6.4' of git://git.linaro.org/landing-teams/working/fujitsu/integration:
        dt-bindings: mailbox: qcom: add compatible for IPQ9574 SoC
        mailbox: qcom-apcs-ipc: do not grow the of_device_id
        dt-bindings: mailbox: qcom,apcs-kpss-global: use fallbacks for few variants
        dt-bindings: mailbox: mediatek,gce-mailbox: Add support for MT6795
        mailbox: mpfs: convert SOC_MICROCHIP_POLARFIRE to ARCH_MICROCHIP_POLARFIRE
        mailbox: bcm-pdc: drop of_match_ptr for ID table
        mailbox: rockchip: drop of_match_ptr for ID table
        mailbox: mailbox-test: Fix potential double-free in mbox_test_message_write()
        mailbox: mailbox-test: Explicitly include header for spinlock support
        mailbox: Use of_property_read_bool() for boolean properties
        mailbox: pcc: Use mbox_bind_client
        mailbox: omap: Use mbox_bind_client
        mailbox: Allow direct registration to a channel
      1c1094e4
    • Linus Torvalds's avatar
      Merge tag 'for-6.4/io_uring-2023-05-07' of git://git.kernel.dk/linux · 03e5cb7b
      Linus Torvalds authored
      Pull more io_uring updates from Jens Axboe:
       "Nothing major in here, just two different parts:
      
         - A small series from Breno that enables passing the full SQE down
           for ->uring_cmd().
      
           This is a prerequisite for enabling full network socket operations.
           Queued up a bit late because of some stylistic concerns that got
           resolved, would be nice to have this in 6.4-rc1 so the dependent
           work will be easier to handle for 6.5.
      
         - Fix for the huge page coalescing, which was a regression introduced
           in the 6.3 kernel release (Tobias)"
      
      * tag 'for-6.4/io_uring-2023-05-07' of git://git.kernel.dk/linux:
        io_uring: Remove unnecessary BUILD_BUG_ON
        io_uring: Pass whole sqe to commands
        io_uring: Create a helper to return the SQE size
        io_uring/rsrc: check for nonconsecutive pages
      03e5cb7b
  5. May 07, 2023