blk-mq gives the block layer a software staging queue per CPU and a hardware dispatch queue per device queue, with the I/O scheduler between them. A request takes the shortest path — straight to the hardware queue — unless a scheduler is attached or it can be merged, in which case it is staged in software first. That scheduler is the only place in the stack where the kernel decides the order your requests reach the device.

Ordering is worth a great deal on a rotating disk and almost nothing on flash, but the scheduler has a second job that survives the move to NVMe: deciding who waits. Those jobs pull in different directions — reordering for head efficiency is indifferent to fairness, and fairness costs seeks — which is why the four schedulers are matched to a device and a workload rather than ranked.

The head is the budget

On a 7200 RPM disk, rotation costs 4.17 ms per request on average and transferring a 4 KiB block costs 0.026 ms, so almost all service time is seek and rotation. The fixture replays 2000 random 4 KiB reads across 100,000 tracks: in arrival order the head travels 67,907,411 tracks and the run takes 21,954 ms, or 91 IOPS; in ascending track order it travels 99,977 tracks and takes 8,645 ms, or 231 IOPS. Same requests and device: 679 times less head travel, 2.5 times the throughput.

Seek is 62 percent of service time in the arrival-order run and 3 percent in the sorted run, while rotation and transfer stay fixed at 4.19 ms either way. On NVMe the same requests take about 40 ms with no seek at all, so sorting buys nothing.

What each scheduler is for

none is first-come-first-served: right for a deep internal queue with no contention, wrong on a rotator, where the seek component stays untouched. mq-deadline sorts by sector and adds per-direction expiry — reads expire long before writes, each batch capped by fifo_batch — which bounds the tail none leaves unbounded. kyber targets latencies directly, throttling a queue whose requests miss the read or write target (read_lat_nsec, write_lat_nsec), so a bulk writer cannot push interactive reads past the target alone.

bfq is the fairness answer: a proportional-share scheduler with cgroup support that distributes device throughput — not just device time — among processes and groups in proportion to their weights. It privileges latency over throughput by default and idles the device, which helps on rotational media but can cost throughput on flash. Read the active scheduler with cat /sys/block/<dev>/queue/scheduler, which prints the list with the current one in brackets, and change it with echo bfq > /sys/block/sda/queue/scheduler; make that survive reboot with a udev rule, since the attribute is recreated per device.

Fairness is a different question from order

With a 90/10 issue mix between a bulk hog and a light interactive client, arrival order gives the light client 0.099 of the service and an average wait position of 1901 requests — not starved, drowned. Under per-group fair queueing it gets 0.5 of the service regardless of the mix, at the hog’s expense. Only the arbitration changed.

So on flash the choice is usually bfq for isolation or mq-deadline for bounded tails, and none only when one workload owns the device. On a rotator the sorted order is the larger win. In cgroup v2 terms, io.weight gives a group a share and io.max a ceiling; both only matter if the scheduler enforces the order.

When it breaks

The classic failure is a rotator running [none], usually because a host was cloned from an NVMe image or a udev rule matched sd* while the device is nvme0n1. The signature is %util near 100 percent with r/s low and await in the tens of milliseconds — the device is busy, not saturated, and it is busy seeking. Confirm with cat /sys/block/<dev>/queue/rotational: 1 with no reordering scheduler is the bug.

The second failure is a latency collapse with healthy averages. A backup job or a dd on the same device starves interactive reads, so p99 explodes while the mean barely moves. none and mq-deadline will not protect the interactive client; bfq or a cgroup io.max will. On flash, watch slice_idle: idling to preserve sequentiality is a throughput loss when there is no head to preserve.

Check it quickly

Read what the device is and which scheduler it is running:

for d in /sys/block/sd* /sys/block/nvme*n*; do echo "$d rot=$(cat $d/queue/rotational) sched=$(cat $d/queue/scheduler)"; done
cat /sys/block/sda/queue/iosched/read_expire /sys/block/sda/queue/iosched/fifo_batch
iostat -x 2 5
echo mq-deadline > /sys/block/sda/queue/scheduler

The first line pairs each device’s rotational flag with its active scheduler in brackets; rot=1 next to [none] is the failure signature, rot=0 next to [none] is normal. The second shows mq-deadline’s expiry and batch tunables. iostat -x distinguishes saturated from merely busy: high %util with low r/s and high await means queueing and seeking.

Sources