Io · XB Field Notes

iotop shows which process is burning the disks

Wed, 15 Mar 2028 00:00:00 +0000

A disk that spikes with no obvious writer is hard to chase; iotop maps throughput straight back to the task doing the damage. System-wide tools show the symptom — high await, saturated utilization — but never the suspect: fifty processes share the disk and the average blames all of them equally. The question that ends the investigation is always per-process — who, how much, since when — and iotop answers exactly that, live, from kernel counters no application can hide from.

Keep it beside the system-wide view rather than instead of it. iostat or vmstat says the disk is the bottleneck; iotop names the process making it so. Together they compress a storage mystery into a two-command diagnosis: confirm the resource, identify the consumer.

Per-process counters

iotop reads per-process IO from the kernel, so live read and write rates map to a pid rather than a nameless load average. Run it with -o to show only processes that are actually doing IO. The default view splits reads from writes and shows swap-in percentage alongside throughput, which separates the genuinely disk-hungry from the merely present — a process with high DISK READ but zero writes is scanning, one with writes is producing, and the distinction decides whether you throttle, relocate, or kill.

Note what the counters actually measure: bytes passing through the IO layer per process, including buffered writes not yet on disk. A process can show huge write rates while the disk looks idle (page cache absorbing) or modest rates while the disk saturates (many small synchronous writes). Read iotop together with device utilization: process-side blame plus device-side saturation is the complete picture, and either half alone misleads.

See whatever sweeps the disk

Sort by bandwidth and the offender stands out, whether it is a log hoarder, an indexer, or a runaway backup. Follow the pid to the exact process (and often the file) before acting. The usual lineup: an unrotated log written at debug level, a reindex job scheduled against peak hours, a backup writing to the same spindle it reads from, a container layer unpack looping on a corrupt image. Each has a different fix — rotate, reschedule, separate, prune — and killing the process treats the symptom while the cause reschedules itself for tomorrow.

Resolve the pid to its full context before touching anything: command line, working directory, open files via lsof, parent process. A java process burning writes might be ten applications behind one PID namespace; the fix belongs to the right one. Attribution first, action second — every storage war story that ends in regret skipped the first step.

Watch before you act

iotop observes rather than changes anything, so you can investigate the culprit before deciding what to throttle, move, or kill. Pair it with strace or lsof on the pid to pin the file and the syscall. The combination is decisive: iotop says process 18412 writes 400 MB/s, lsof says the writes go to one growing log file, strace shows O_APPEND writes of 4 KB — diagnosis complete, and the fix (log level, rotation, async appender) is obvious instead of debated.

Record the baseline while healthy so the incident comparison means something. A five-minute iotop batch capture on a normal day — -b -n 60 to a file — gives you “normal for this host” in numbers: which processes usually lead, what rates are ordinary. Without it, every spike looks unprecedented; with it, most incidents classify in seconds as “the indexer, again, an hour early.”

When it breaks

The classic meta-failure is permission blindness: run without privileges and iotop shows question marks instead of process details, or hides other users’ processes entirely — and the actual offender, running as another user, stays invisible. Run with the necessary privileges (root or the right capabilities) from the start, or the tool exonerates exactly the processes you cannot see. An investigation that “found nothing” with unprivileged iotop found nothing about your own processes only.

The second failure is sampling illusion on bursty IO: a process doing violent one-second bursts every minute looks innocent in a 5-second average and guilty in a 1-second one, or vice versa. Vary the delay interval when the numbers and the symptoms disagree, and prefer accumulated totals (-a) over instantaneous rates for “who hurt the disk over the last hour” questions. The right aggregation answers the question asked; the default answers whichever question its interval assumes.

Check it quickly

Capture who does IO, with context, non-interactively:

iotop -b -n 5 -o -d 2 | head -n 30
pid=$(iotop -b -n 1 -o | awk 'NR>5 {print $1}' | head -n 1); ls -l /proc/$pid/exe; cat /proc/$pid/cmdline | tr '\0' ' '
iostat -x 1 3 | grep -E "Device|sd|nvme"

The batch capture ranks offenders without a terminal UI, the pid resolution names the top writer’s binary, and iostat confirms the device side is actually saturated.

Sources

iotop man page, Official iotop reference, checked 2026-06-01.
vmstat man page, documents system-wide CPU and IO-wait reporting, checked 2026-09-05.

Disk schedulers order I/O to move the head efficiently

Tue, 09 Mar 2027 00:00:00 +0000

blk-mq gives the block layer a software staging queue per CPU and a hardware dispatch queue per device queue, with the I/O scheduler between them. A request takes the shortest path — straight to the hardware queue — unless a scheduler is attached or it can be merged, in which case it is staged in software first. That scheduler is the only place in the stack where the kernel decides the order your requests reach the device.

Ordering is worth a great deal on a rotating disk and almost nothing on flash, but the scheduler has a second job that survives the move to NVMe: deciding who waits. Those jobs pull in different directions — reordering for head efficiency is indifferent to fairness, and fairness costs seeks — which is why the four schedulers are matched to a device and a workload rather than ranked.

The head is the budget

On a 7200 RPM disk, rotation costs 4.17 ms per request on average and transferring a 4 KiB block costs 0.026 ms, so almost all service time is seek and rotation. The fixture replays 2000 random 4 KiB reads across 100,000 tracks: in arrival order the head travels 67,907,411 tracks and the run takes 21,954 ms, or 91 IOPS; in ascending track order it travels 99,977 tracks and takes 8,645 ms, or 231 IOPS. Same requests and device: 679 times less head travel, 2.5 times the throughput.

Seek is 62 percent of service time in the arrival-order run and 3 percent in the sorted run, while rotation and transfer stay fixed at 4.19 ms either way. On NVMe the same requests take about 40 ms with no seek at all, so sorting buys nothing.

What each scheduler is for

none is first-come-first-served: right for a deep internal queue with no contention, wrong on a rotator, where the seek component stays untouched. mq-deadline sorts by sector and adds per-direction expiry — reads expire long before writes, each batch capped by fifo_batch — which bounds the tail none leaves unbounded. kyber targets latencies directly, throttling a queue whose requests miss the read or write target (read_lat_nsec, write_lat_nsec), so a bulk writer cannot push interactive reads past the target alone.

bfq is the fairness answer: a proportional-share scheduler with cgroup support that distributes device throughput — not just device time — among processes and groups in proportion to their weights. It privileges latency over throughput by default and idles the device, which helps on rotational media but can cost throughput on flash. Read the active scheduler with cat /sys/block/<dev>/queue/scheduler, which prints the list with the current one in brackets, and change it with echo bfq > /sys/block/sda/queue/scheduler; make that survive reboot with a udev rule, since the attribute is recreated per device.

Fairness is a different question from order

With a 90/10 issue mix between a bulk hog and a light interactive client, arrival order gives the light client 0.099 of the service and an average wait position of 1901 requests — not starved, drowned. Under per-group fair queueing it gets 0.5 of the service regardless of the mix, at the hog’s expense. Only the arbitration changed.

So on flash the choice is usually bfq for isolation or mq-deadline for bounded tails, and none only when one workload owns the device. On a rotator the sorted order is the larger win. In cgroup v2 terms, io.weight gives a group a share and io.max a ceiling; both only matter if the scheduler enforces the order.

When it breaks

The classic failure is a rotator running [none], usually because a host was cloned from an NVMe image or a udev rule matched sd* while the device is nvme0n1. The signature is %util near 100 percent with r/s low and await in the tens of milliseconds — the device is busy, not saturated, and it is busy seeking. Confirm with cat /sys/block/<dev>/queue/rotational: 1 with no reordering scheduler is the bug.

The second failure is a latency collapse with healthy averages. A backup job or a dd on the same device starves interactive reads, so p99 explodes while the mean barely moves. none and mq-deadline will not protect the interactive client; bfq or a cgroup io.max will. On flash, watch slice_idle: idling to preserve sequentiality is a throughput loss when there is no head to preserve.

Check it quickly

Read what the device is and which scheduler it is running:

for d in /sys/block/sd* /sys/block/nvme*n*; do echo "$d rot=$(cat $d/queue/rotational) sched=$(cat $d/queue/scheduler)"; done
cat /sys/block/sda/queue/iosched/read_expire /sys/block/sda/queue/iosched/fifo_batch
iostat -x 2 5
echo mq-deadline > /sys/block/sda/queue/scheduler

The first line pairs each device’s rotational flag with its active scheduler in brackets; rot=1 next to [none] is the failure signature, rot=0 next to [none] is normal. The second shows mq-deadline’s expiry and batch tunables. iostat -x distinguishes saturated from merely busy: high %util with low r/s and high await means queueing and seeking.

Sources

Switching Scheduler, the scheduler names, sysfs paths and how to change them, checked 2026-09-16.
Multi-Queue Block IO Queueing Mechanism (blk-mq), software staging queues versus hardware dispatch queues, checked 2026-09-16.
BFQ (Budget Fair Queueing), proportional-share behaviour, cgroup support and the idling trade-off, checked 2026-09-16.
Kyber I/O scheduler tunables, the read and write latency targets Kyber throttles against, checked 2026-09-16.