High iowait in TrueNAS server

I recently ran into a performance issue on my TrueNAS SCALE 25.10.1 system where the server felt sluggish despite low CPU usage. The system was running Docker-based applications, and at first glance nothing obvious looked wrong. The real problem turned out to be high iowait.

What iowait actually means

In Linux, iowait represents the percentage of time the CPU is idle while waiting for I/O operations (usually disk). High iowait doesn’t mean the CPU is busy — it means the CPU is stuck waiting on storage.

In top, this appears as wa:

%Cpu(s):  1.8 us,  1.7 sy,  0.0 ni, 95.5 id,  0.2 wa,  0.0 hi,  0.8 si,  0.0 st

Under normal conditions, iowait should stay very low (usually under 1–2%). When it starts climbing higher, the system can feel slow even if CPU usage looks fine.

Confirming the issue with iostat

To get a clearer picture, I used iostat, which shows per-disk activity and latency:

iostat -x 1

This immediately showed the problem. One or more disks had:

  • Very high %util (near or at 100%)
  • Elevated await times
  • Consistent read/write pressure

At that point it was clear the bottleneck was storage I/O, not CPU or memory.

Tracking it down to Docker services

This system runs several Docker-based services. Using top alongside iostat, I noticed disk activity drop immediately when certain services were stopped.

In particular, high I/O was coming from applications that:

  • Continuously read/write large files
  • Perform frequent metadata operations
  • Maintain large active datasets

Examples included downloaders, media managers, and backup-related containers.

Stopping services to confirm

To confirm the cause, I stopped Docker services one at a time and watched disk metrics:

iostat -x 1

Each time a heavy I/O service was stopped, iowait dropped immediately. Once the worst offender was stopped, iowait returned to normal levels and the system became responsive again.

Why the system looked “fine” at first

This was tricky because:

  • CPU usage was low
  • Memory usage looked reasonable
  • The web UI was responsive but sluggish

Without checking iostat, it would have been easy to misdiagnose this as a CPU or RAM issue.

Lessons learned

  • High iowait can cripple performance even when CPU is idle
  • top alone is not enough — use iostat -x
  • Docker workloads can silently saturate disks
  • Stopping services one by one is an effective diagnostic technique

Final takeaway

On TrueNAS SCALE 25.10.1 with Docker, high iowait was the real cause of my performance issues. The fix wasn’t a reboot, more CPU, or more RAM — it was identifying and controlling disk-heavy services.

If your TrueNAS server feels slow but CPU usage looks fine, check iowait and run iostat. The disk may be the real bottleneck.


Leave a comment

Discover more from /root

Subscribe now to keep reading and get access to the full archive.

Continue reading