iTranslated by AI
CPU Screaming While Waiting? Uncovering the True Nature of IOWait Through Alert Design
Introduction
"IOWait?? Do I really need this alert? And why does wait time affect CPU utilization in the first place?"
Have you ever thought this while reviewing your alerts? You understand Idle and such, but you've never seen it in a Fire state, and you have no idea what the correct threshold should be.
When you consider the von Neumann computer architecture, it feels like a waste to simply view this as idle time. Along with alert design, let's organize this a bit.
What is IOWait?
IOWait stands for "Input/Output Wait." It's the time the CPU spends waiting for I/O operations to complete. You can kind of guess this from the name itself.
Examples:
- Reading a file from disk
- Writing a file to disk
- Network communication
- Writing data to the playback buffer for audio output
- Transferring to the frame buffer (GPU memory) to render to the display
- Accessing a USB drive
A modern 1GHz 64-bit CPU can execute 1 billion instructions per second (1 cycle: 1 nanosecond). In contrast, an old-school HDD takes several milliseconds to read data. To the CPU, I/O is that slow.
Why is it discussed in the context of CPU usage?
"IOWait is about I/O wait time, so why is it included in CPU usage?"
That's what I thought next. It seems this is related to the design philosophy of the Linux kernel. (On Mac, both are combined into idle.)
CPU usage represents how time is spent↓
CPU time = User + System + IOWait + Idle + Others
"So what is this time?" you might wonder. In fact, the CPU divides each second into tiny slices and records what each slice was used for!
Breakdown of CPU time:
- User: Time when applications are doing heavy computation (e.g., Excel calculations, game rendering)
- System: Time when the OS is doing background management work (e.g., memory management, file system operations)
- IOWait: Time when there's work to do, but the CPU is waiting for I/O
- Idle: Peaceful time when there's really nothing to do (e.g., no applications running)
As an example with the top command:
%Cpu(s): 5.2 us, 2.1 sy, 0.0 ni, 85.4 id, 7.3 wa
This "wa" is IOWait. The notation "Cpu(s)" also makes it clear that it represents a breakdown of CPU time.
The "If you're waiting, why not do another job?" problem
This seems to be the key point.
Since this is a multitasking OS, if another process can run, the OS will switch to it. However, if "all CPU cores are waiting for I/O," or "there are no other tasks that can run," or "there are dependencies between threads/processes," the CPU remains idle. This idle time accumulates as "iowait."
At that point, the OS judges it as "I want to work but I'm idle because I/O is slow," not just "idle," so it's counted as iowait.
When IOWait is High vs. Low
When IOWait is High (20% or more)
Understanding when IOWait is high is relatively straightforward. It means there are dependencies between processes or threads, so they aren't waiting for I/O, but there's no runnable work left—it's a massive traffic jam! The CPU is screaming "I want to work but I can't!"
Symptoms look like this:
- Applications feel sluggish (even though process load might not be that high)
- Database queries are slow
- File copying never seems to finish
Common suspects:
- Log file bloat: Excessive writes cause chaos
- Database runaway: Full table scans wreak havoc
- Backup processes: Sneaking away with large amounts of data late at night
- Memory shortage and swapping: Running out of memory and spilling to disk
When IOWait is Low (5% or less)
When IOWait is low, it's a sign the system is running smoothly.
However, you shouldn't let your guard down:
- If other CPU metrics (User or System) are high, there might be a different culprit lurking
- If it's too low, it could mean almost no I/O operations are occurring (which is also a concern)
Summary
I've tried to organize how IOWait should be discussed in alert design. From now on, I'll keep thinking about the meaning of metrics.
I need to relearn information science...
Discussion