1. Mar 18, 2022
    • Ming Lei's avatar
      block: avoid use-after-free on throttle data · ee37eddb
      Ming Lei authored
      In throtl_pending_timer_fn(), request queue is retrieved from throttle
      data. And tg's pending timer is deleted synchronously when releasing the
      associated blkg, at that time, throttle data may have been freed since
      commit 1059699f
      
       ("block: move blkcg initialization/destroy into disk
      allocation/release handler") moves freeing q->td to disk_release() from
      blk_release_queue(). So use-after-free on q->td in throtl_pending_timer_fn
      can be triggered.
      
      Fixes the issue by:
      
      - do nothing in case that disk is released, when there isn't any bio to
        dispatch
      
      - retrieve request queue from blkg instead of throttle data for
      non top-level pending timer.
      
      Signed-off-by: default avatarMing Lei <ming.lei@redhat.com>
      Link: https://lore.kernel.org/r/20220318130144.1066064-2-ming.lei@redhat.com
      
      
      Signed-off-by: default avatarJens Axboe <axboe@kernel.dk>
      ee37eddb
    • Shin'ichiro Kawasaki's avatar
      block: limit request dispatch loop duration · 572299f0
      Shin'ichiro Kawasaki authored
      When IO requests are made continuously and the target block device
      handles requests faster than request arrival, the request dispatch loop
      keeps on repeating to dispatch the arriving requests very long time,
      more than a minute. Since the loop runs as a workqueue worker task, the
      very long loop duration triggers workqueue watchdog timeout and BUG [1].
      
      To avoid the very long loop duration, break the loop periodically. When
      opportunity to dispatch requests still exists, check need_resched(). If
      need_resched() returns true, the dispatch loop already consumed its time
      slice, then reschedule the dispatch work and break the loop. With heavy
      IO load, need_resched() does not return true for 20~30 seconds. To cover
      such case, check time spent in the dispatch loop with jiffies. If more
      than 1 second is spent, reschedule the dispatch work and break the loop.
      
      [1]
      
      [  609.691437] BUG: workqueue lockup - pool cpus=10 node=1 flags=0x0 nice=-20 stuck for 35s!
      [  609.701820] Showing busy workqueues and worker pools:
      [  609.707915] workqueue events: flags=0x0
      [  609.712615]   pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2
      [  609.712626]     pending: drm_fb_helper_damage_work [drm_kms_helper]
      [  609.712687] workqueue events_freezable: flags=0x4
      [  609.732943]   pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2
      [  609.732952]     pending: pci_pme_list_scan
      [  609.732968] workqueue events_power_efficient: flags=0x80
      [  609.751947]   pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2
      [  609.751955]     pending: neigh_managed_work
      [  609.752018] workqueue kblockd: flags=0x18
      [  609.769480]   pwq 21: cpus=10 node=1 flags=0x0 nice=-20 active=3/256 refcnt=4
      [  609.769488]     in-flight: 1020:blk_mq_run_work_fn
      [  609.769498]     pending: blk_mq_timeout_work, blk_mq_run_work_fn
      [  609.769744] pool 21: cpus=10 node=1 flags=0x0 nice=-20 hung=35s workers=2 idle: 67
      [  639.899730] BUG: workqueue lockup - pool cpus=10 node=1 flags=0x0 nice=-20 stuck for 66s!
      [  639.909513] Showing busy workqueues and worker pools:
      [  639.915404] workqueue events: flags=0x0
      [  639.920197]   pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2
      [  639.920215]     pending: drm_fb_helper_damage_work [drm_kms_helper]
      [  639.920365] workqueue kblockd: flags=0x18
      [  639.939932]   pwq 21: cpus=10 node=1 flags=0x0 nice=-20 active=3/256 refcnt=4
      [  639.939942]     in-flight: 1020:blk_mq_run_work_fn
      [  639.939955]     pending: blk_mq_timeout_work, blk_mq_run_work_fn
      [  639.940212] pool 21: cpus=10 node=1 flags=0x0 nice=-20 hung=66s workers=2 idle: 67
      
      Fixes: 6e6fcbc2
      
       ("blk-mq: support batching dispatch in case of io")
      Signed-off-by: default avatarShin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
      Cc: stable@vger.kernel.org # v5.10+
      Link: https://lore.kernel.org/linux-block/20220310091649.zypaem5lkyfadymg@shindev/
      Link: https://lore.kernel.org/r/20220318022641.133484-1-shinichiro.kawasaki@wdc.com
      
      
      Signed-off-by: default avatarJens Axboe <axboe@kernel.dk>
      572299f0
  2. Mar 16, 2022
  3. Mar 15, 2022
    • Tejun Heo's avatar
      block: don't merge across cgroup boundaries if blkcg is enabled · 6b2b0459
      Tejun Heo authored
      
      
      blk-iocost and iolatency are cgroup aware rq-qos policies but they didn't
      disable merges across different cgroups. This obviously can lead to
      accounting and control errors but more importantly to priority inversions -
      e.g. an IO which belongs to a higher priority cgroup or IO class may end up
      getting throttled incorrectly because it gets merged to an IO issued from a
      low priority cgroup.
      
      Fix it by adding blk_cgroup_mergeable() which is called from merge paths and
      rejects cross-cgroup and cross-issue_as_root merges.
      
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      Fixes: d7067512 ("block: introduce blk-iolatency io controller")
      Cc: stable@vger.kernel.org # v4.19+
      Cc: Josef Bacik <jbacik@fb.com>
      Link: https://lore.kernel.org/r/Yi/eE/6zFNyWJ+qd@slm.duckdns.org
      
      
      Signed-off-by: default avatarJens Axboe <axboe@kernel.dk>
      6b2b0459
    • Tejun Heo's avatar
      block: fix rq-qos breakage from skipping rq_qos_done_bio() · aa1b46dc
      Tejun Heo authored
      a647a524
      
       ("block: don't call rq_qos_ops->done_bio if the bio isn't
      tracked") made bio_endio() skip rq_qos_done_bio() if BIO_TRACKED is not set.
      While this fixed a potential oops, it also broke blk-iocost by skipping the
      done_bio callback for merged bios.
      
      Before, whether a bio goes through rq_qos_throttle() or rq_qos_merge(),
      rq_qos_done_bio() would be called on the bio on completion with BIO_TRACKED
      distinguishing the former from the latter. rq_qos_done_bio() is not called
      for bios which wenth through rq_qos_merge(). This royally confuses
      blk-iocost as the merged bios never finish and are considered perpetually
      in-flight.
      
      One reliably reproducible failure mode is an intermediate cgroup geting
      stuck active preventing its children from being activated due to the
      leaf-only rule, leading to loss of control. The following is from
      resctl-bench protection scenario which emulates isolating a web server like
      workload from a memory bomb run on an iocost configuration which should
      yield a reasonable level of protection.
      
        # cat /sys/block/nvme2n1/device/model
        Samsung SSD 970 PRO 512GB
        # cat /sys/fs/cgroup/io.cost.model
        259:0 ctrl=user model=linear rbps=834913556 rseqiops=93622 rrandiops=102913 wbps=618985353 wseqiops=72325 wrandiops=71025
        # cat /sys/fs/cgroup/io.cost.qos
        259:0 enable=1 ctrl=user rpct=95.00 rlat=18776 wpct=95.00 wlat=8897 min=60.00 max=100.00
        # resctl-bench -m 29.6G -r out.json run protection::scenario=mem-hog,loops=1
        ...
        Memory Hog Summary
        ==================
      
        IO Latency: R p50=242u:336u/2.5m p90=794u:1.4m/7.5m p99=2.7m:8.0m/62.5m max=8.0m:36.4m/350m
                    W p50=221u:323u/1.5m p90=709u:1.2m/5.5m p99=1.5m:2.5m/9.5m max=6.9m:35.9m/350m
      
        Isolation and Request Latency Impact Distributions:
      
                      min   p01   p05   p10   p25   p50   p75   p90   p95   p99   max  mean stdev
        isol%       15.90 15.90 15.90 40.05 57.24 59.07 60.01 74.63 74.63 90.35 90.35 58.12 15.82
        lat-imp%        0     0     0     0     0  4.55 14.68 15.54 233.5 548.1 548.1 53.88 143.6
      
        Result: isol=58.12:15.82% lat_imp=53.88%:143.6 work_csv=100.0% missing=3.96%
      
      The isolation result of 58.12% is close to what this device would show
      without any IO control.
      
      Fix it by introducing a new flag BIO_QOS_MERGED to mark merged bios and
      calling rq_qos_done_bio() on them too. For consistency and clarity, rename
      BIO_TRACKED to BIO_QOS_THROTTLED. The flag checks are moved into
      rq_qos_done_bio() so that it's next to the code paths that set the flags.
      
      With the patch applied, the above same benchmark shows:
      
        # resctl-bench -m 29.6G -r out.json run protection::scenario=mem-hog,loops=1
        ...
        Memory Hog Summary
        ==================
      
        IO Latency: R p50=123u:84.4u/985u p90=322u:256u/2.5m p99=1.6m:1.4m/9.5m max=11.1m:36.0m/350m
                    W p50=429u:274u/995u p90=1.7m:1.3m/4.5m p99=3.4m:2.7m/11.5m max=7.9m:5.9m/26.5m
      
        Isolation and Request Latency Impact Distributions:
      
                      min   p01   p05   p10   p25   p50   p75   p90   p95   p99   max  mean stdev
        isol%       84.91 84.91 89.51 90.73 92.31 94.49 96.36 98.04 98.71 100.0 100.0 94.42  2.81
        lat-imp%        0     0     0     0     0  2.81  5.73 11.11 13.92 17.53 22.61  4.10  4.68
      
        Result: isol=94.42:2.81% lat_imp=4.10%:4.68 work_csv=58.34% missing=0%
      
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      Fixes: a647a524
      
       ("block: don't call rq_qos_ops->done_bio if the bio isn't tracked")
      Cc: stable@vger.kernel.org # v5.15+
      Cc: Ming Lei <ming.lei@redhat.com>
      Cc: Yu Kuai <yukuai3@huawei.com>
      Reviewed-by: default avatarMing Lei <ming.lei@redhat.com>
      Link: https://lore.kernel.org/r/Yi7rdrzQEHjJLGKB@slm.duckdns.org
      
      
      Signed-off-by: default avatarJens Axboe <axboe@kernel.dk>
      aa1b46dc
  4. Mar 12, 2022
  5. Mar 09, 2022
  6. Mar 07, 2022
  7. Mar 05, 2022
    • Zhang Wensheng's avatar
      bfq: fix use-after-free in bfq_dispatch_request · ab552fcb
      Zhang Wensheng authored
      
      
      KASAN reports a use-after-free report when doing normal scsi-mq test
      
      [69832.239032] ==================================================================
      [69832.241810] BUG: KASAN: use-after-free in bfq_dispatch_request+0x1045/0x44b0
      [69832.243267] Read of size 8 at addr ffff88802622ba88 by task kworker/3:1H/155
      [69832.244656]
      [69832.245007] CPU: 3 PID: 155 Comm: kworker/3:1H Not tainted 5.10.0-10295-g576c6382529e #8
      [69832.246626] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.14.0-0-g155821a1990b-prebuilt.qemu.org 04/01/2014
      [69832.249069] Workqueue: kblockd blk_mq_run_work_fn
      [69832.250022] Call Trace:
      [69832.250541]  dump_stack+0x9b/0xce
      [69832.251232]  ? bfq_dispatch_request+0x1045/0x44b0
      [69832.252243]  print_address_description.constprop.6+0x3e/0x60
      [69832.253381]  ? __cpuidle_text_end+0x5/0x5
      [69832.254211]  ? vprintk_func+0x6b/0x120
      [69832.254994]  ? bfq_dispatch_request+0x1045/0x44b0
      [69832.255952]  ? bfq_dispatch_request+0x1045/0x44b0
      [69832.256914]  kasan_report.cold.9+0x22/0x3a
      [69832.257753]  ? bfq_dispatch_request+0x1045/0x44b0
      [69832.258755]  check_memory_region+0x1c1/0x1e0
      [69832.260248]  bfq_dispatch_request+0x1045/0x44b0
      [69832.261181]  ? bfq_bfqq_expire+0x2440/0x2440
      [69832.262032]  ? blk_mq_delay_run_hw_queues+0xf9/0x170
      [69832.263022]  __blk_mq_do_dispatch_sched+0x52f/0x830
      [69832.264011]  ? blk_mq_sched_request_inserted+0x100/0x100
      [69832.265101]  __blk_mq_sched_dispatch_requests+0x398/0x4f0
      [69832.266206]  ? blk_mq_do_dispatch_ctx+0x570/0x570
      [69832.267147]  ? __switch_to+0x5f4/0xee0
      [69832.267898]  blk_mq_sched_dispatch_requests+0xdf/0x140
      [69832.268946]  __blk_mq_run_hw_queue+0xc0/0x270
      [69832.269840]  blk_mq_run_work_fn+0x51/0x60
      [69832.278170]  process_one_work+0x6d4/0xfe0
      [69832.278984]  worker_thread+0x91/0xc80
      [69832.279726]  ? __kthread_parkme+0xb0/0x110
      [69832.280554]  ? process_one_work+0xfe0/0xfe0
      [69832.281414]  kthread+0x32d/0x3f0
      [69832.282082]  ? kthread_park+0x170/0x170
      [69832.282849]  ret_from_fork+0x1f/0x30
      [69832.283573]
      [69832.283886] Allocated by task 7725:
      [69832.284599]  kasan_save_stack+0x19/0x40
      [69832.285385]  __kasan_kmalloc.constprop.2+0xc1/0xd0
      [69832.286350]  kmem_cache_alloc_node+0x13f/0x460
      [69832.287237]  bfq_get_queue+0x3d4/0x1140
      [69832.287993]  bfq_get_bfqq_handle_split+0x103/0x510
      [69832.289015]  bfq_init_rq+0x337/0x2d50
      [69832.289749]  bfq_insert_requests+0x304/0x4e10
      [69832.290634]  blk_mq_sched_insert_requests+0x13e/0x390
      [69832.291629]  blk_mq_flush_plug_list+0x4b4/0x760
      [69832.292538]  blk_flush_plug_list+0x2c5/0x480
      [69832.293392]  io_schedule_prepare+0xb2/0xd0
      [69832.294209]  io_schedule_timeout+0x13/0x80
      [69832.295014]  wait_for_common_io.constprop.1+0x13c/0x270
      [69832.296137]  submit_bio_wait+0x103/0x1a0
      [69832.296932]  blkdev_issue_discard+0xe6/0x160
      [69832.297794]  blk_ioctl_discard+0x219/0x290
      [69832.298614]  blkdev_common_ioctl+0x50a/0x1750
      [69832.304715]  blkdev_ioctl+0x470/0x600
      [69832.305474]  block_ioctl+0xde/0x120
      [69832.306232]  vfs_ioctl+0x6c/0xc0
      [69832.306877]  __se_sys_ioctl+0x90/0xa0
      [69832.307629]  do_syscall_64+0x2d/0x40
      [69832.308362]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
      [69832.309382]
      [69832.309701] Freed by task 155:
      [69832.310328]  kasan_save_stack+0x19/0x40
      [69832.311121]  kasan_set_track+0x1c/0x30
      [69832.311868]  kasan_set_free_info+0x1b/0x30
      [69832.312699]  __kasan_slab_free+0x111/0x160
      [69832.313524]  kmem_cache_free+0x94/0x460
      [69832.314367]  bfq_put_queue+0x582/0x940
      [69832.315112]  __bfq_bfqd_reset_in_service+0x166/0x1d0
      [69832.317275]  bfq_bfqq_expire+0xb27/0x2440
      [69832.318084]  bfq_dispatch_request+0x697/0x44b0
      [69832.318991]  __blk_mq_do_dispatch_sched+0x52f/0x830
      [69832.319984]  __blk_mq_sched_dispatch_requests+0x398/0x4f0
      [69832.321087]  blk_mq_sched_dispatch_requests+0xdf/0x140
      [69832.322225]  __blk_mq_run_hw_queue+0xc0/0x270
      [69832.323114]  blk_mq_run_work_fn+0x51/0x60
      [69832.323942]  process_one_work+0x6d4/0xfe0
      [69832.324772]  worker_thread+0x91/0xc80
      [69832.325518]  kthread+0x32d/0x3f0
      [69832.326205]  ret_from_fork+0x1f/0x30
      [69832.326932]
      [69832.338297] The buggy address belongs to the object at ffff88802622b968
      [69832.338297]  which belongs to the cache bfq_queue of size 512
      [69832.340766] The buggy address is located 288 bytes inside of
      [69832.340766]  512-byte region [ffff88802622b968, ffff88802622bb68)
      [69832.343091] The buggy address belongs to the page:
      [69832.344097] page:ffffea0000988a00 refcount:1 mapcount:0 mapping:0000000000000000 index:0xffff88802622a528 pfn:0x26228
      [69832.346214] head:ffffea0000988a00 order:2 compound_mapcount:0 compound_pincount:0
      [69832.347719] flags: 0x1fffff80010200(slab|head)
      [69832.348625] raw: 001fffff80010200 ffffea0000dbac08 ffff888017a57650 ffff8880179fe840
      [69832.354972] raw: ffff88802622a528 0000000000120008 00000001ffffffff 0000000000000000
      [69832.356547] page dumped because: kasan: bad access detected
      [69832.357652]
      [69832.357970] Memory state around the buggy address:
      [69832.358926]  ffff88802622b980: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
      [69832.360358]  ffff88802622ba00: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
      [69832.361810] >ffff88802622ba80: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
      [69832.363273]                       ^
      [69832.363975]  ffff88802622bb00: fb fb fb fb fb fb fb fb fb fb fb fb fb fc fc fc
      [69832.375960]  ffff88802622bb80: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
      [69832.377405] ==================================================================
      
      In bfq_dispatch_requestfunction, it may have function call:
      
      bfq_dispatch_request
      	__bfq_dispatch_request
      		bfq_select_queue
      			bfq_bfqq_expire
      				__bfq_bfqd_reset_in_service
      					bfq_put_queue
      						kmem_cache_free
      In this function call, in_serv_queue has beed expired and meet the
      conditions to free. In the function bfq_dispatch_request, the address
      of in_serv_queue pointing to has been released. For getting the value
      of idle_timer_disabled, it will get flags value from the address which
      in_serv_queue pointing to, then the problem of use-after-free happens;
      
      Fix the problem by check in_serv_queue == bfqd->in_service_queue, to
      get the value of idle_timer_disabled if in_serve_queue is equel to
      bfqd->in_service_queue. If the space of in_serv_queue pointing has
      been released, this judge will aviod use-after-free problem.
      And if in_serv_queue may be expired or finished, the idle_timer_disabled
      will be false which would not give effects to bfq_update_dispatch_stats.
      
      Reported-by: default avatarHulk Robot <hulkci@huawei.com>
      Signed-off-by: default avatarZhang Wensheng <zhangwensheng5@huawei.com>
      Link: https://lore.kernel.org/r/20220303070334.3020168-1-zhangwensheng5@huawei.com
      
      
      Signed-off-by: default avatarJens Axboe <axboe@kernel.dk>
      ab552fcb