Slow NVMe disk? How to spot I/O and IOPS bottlenecks on Linux

Why NVMe storage can still bottleneck your system

Modern storage units built on the NVMe protocol revolutionized computing by eliminating the mechanical bottlenecks of legacy hard drives and bypassing traditional SATA busses. In practice, this means our servers can now process hundreds of thousands of operations per second without breaking a sweat. However, even with stratospheric read and write speeds, high-demand applications still suffer from unexplained slowness. Understanding storage subsystem health requires looking far beyond simple free disk space metrics.

When an application hangs waiting for data from persistent memory, the issue is rarely a lack of raw capacity. The real culprit is usually I/O latency, which is the time the system takes to fulfill a read or write request. To diagnose this properly, we must look at fundamental metrics like IOPS (input/output operations per second) and command queue depth. Before blaming application code, it is wise to investigate whether the bottleneck lies in the hardware or controller, something we also explore when looking at how to diagnose slow NVMe SSDs on Linux directly from the terminal.

Monitoring disk pulse with iostat

The fastest and most accessible tool for inspecting disk activity on Linux is part of the sysstat package. The iostat command displays detailed statistics about CPU utilization and the performance of disk partitions and logical units. By running it with interval parameters, we can observe the exact hardware behavior under load. In practice, it gives us a panoramic view of how much time the disk spends processing requests and the average size of accumulated queues.

To start monitoring in real-time with updates every two seconds, simply run the command below in your server terminal. Pay close attention to the await column, which measures the average time in milliseconds that requests take to be served, including queue time and physical processing time by the device.

iostat -xz 2 5

If the await values spike while the utilization percentage (util) hits one hundred percent, you have found a clear indicator of saturation. This means the disk is receiving more requests than it can process concurrently. Keep in mind that parallel temperature issues, as discussed when exploring how to monitor NVMe temperature on Linux via terminal, will also throttle speeds to prevent thermal damage.

Measuring real capacity with stress testing

Often, the technical specifications printed on the NVMe SSD box represent ideal scenarios that rarely happen in the real world. To know if your hardware is delivering what it promises or suffering from wear degradation, the best approach is to run controlled tests. The fio (Flexible I/O Tester) utility is the industry standard for simulating complex workload patterns, ranging from massive sequential access to thousands of simultaneous random reads.

The command below runs a quick random write test with four-kilobyte blocks, simulating the typical behavior of a relational database moving heavy indexes. It uses multiple processes to stress the controller and measure real latency behavior under pressure.

fio --name=randwrite-test --ioengine=libaio --rw=randwrite --bs=4k --numjobs=4 --size=1G --iodepth=64 --runtime=60 --time_based

Analyzing the report generated by fio requires paying special attention to latency distribution expressed in percentiles. If the ninety-ninth percentile (p99) shows response times in the hundreds of milliseconds, your application will certainly experience noticeable stutters for the end user, even if the overall average looks acceptable.

Investigating deep queues and kernel bottlenecks

When iostat points to bottlenecks but synthetic tests show a healthy disk, the issue might lie in how the operating system manages I/O requests. Linux uses software layers in the kernel to organize traffic before delivering it physically to the NVMe bus. Tools like blktrace allow you to trace every block-level I/O event, mapping exactly where delays occur in the software stack.

To capture real-time tracing events on a specific drive and convert them into a readable format, we combine blktrace with the blkparse utility. This helps identify whether the kernel I/O scheduler is queuing requests inefficiently for your specific hardware topology.

blktrace -d /dev/nvme0n1 -o - | blkparse -i -

This meticulous analysis is indispensable in high-density virtualization environments or database clusters where multiple processes compete for the same DMA channels. Identifying whether the bottleneck resides in the driver queue or the controller itself prevents unnecessary hardware replacements. To explore these architectural concepts further in production, see our related technical analysis on Failed motherboard network chip? How to test with a multimeter.

Conclusion

Diagnosing I/O bottlenecks on NVMe drives in Linux ceases to be a mystery when we combine real-time observation with targeted stress testing. The secret lies in looking past theoretical peak speeds and focusing instead on latency consistency and command queue behavior under operational stress. Maintaining preventive monitoring routines ensures your infrastructure remains resilient, fast, and ready to handle traffic spikes without imperceptible degradation.