Check the logs

dmesg -Hw   # Human readable, follow
tail -f /var/log/syslog 
tail -f /var/log/syslog

Check RAM usage. Yes, it's always full, but space in “buff/cache” is free to use

free -m

Show all Processes, sorted by virtual sizes (incl. shared libraries)

ps awwlx --sort=vsz

Load can be CPU bound or Disc bound (or caused by network issues).

Load average ist not about instant CPU usage, it shows ow many processes are waiting to run

If the Load is rising constantly, probably there is a hanging process blocking the disc. Check for: - rsync - happens if the remote is down - sshfs - may not release a mount point, if the remote is down - munin - the disc free plugin hangs, when there are stale mounts - …

watch cat /proc/loadavg

top

sar -u 5

Results: Cpu(s): 11.4%us, 29.6%sy, 0.0%ni, 58.3%id, .7%wa, 0.0%hi, 0.0%si, 0.0%st

IF “wa” I/O wait is high, the load is most probably Disc bound (or RAM)

IF “sy” System is high, it might be, mdadm rund a RAID check

iotop -oP

iostat 

This can really help!

ps -e v

Check which processes are in R or D state:

ps -eo s,user,cmd | grep ^[RD] | sort | uniq -c | sort -nbr | head -20

This outputs the state of each process. Specially D is interesting (but also R):

• D Marks a process in disk (or other short term, uninterruptible) wait.
• R Marks a runnable process

PSN - Linux Process Snapper is another nice tool:

Install from: https://tanelpoder.com/psnapper/

• https://www.linuxjournal.com/article/10688
• https://www.tummy.com/articles/isolating-heavy-load/
• https://tanelpoder.com/posts/high-system-load-low-cpu-utilization-on-linux/