====== Debug ======

Check the logs

  dmesg -Hw   # Human readable, follow
  tail -f /var/log/syslog 
  tail -f /var/log/syslog


===== Out of Memory Problems =====

Check RAM usage. Yes, it's always full, but space in "buff/cache" is free to use
  free -m

Show all Processes, sorted by virtual sizes (incl. shared libraries)
  ps awwlx --sort=vsz



===== High Load =====

Load can be CPU bound or Disc bound (or caused by network issues).

Load average ist not about instant CPU usage, it shows ow many processes are waiting to run

If the Load is rising constantly, probably there is a hanging process blocking the disc. Check for:
- rsync - happens if the remote is down
- sshfs - may not release a mount point, if the remote is down
- munin - the disc free plugin hangs, when there are stale mounts
- ...



=== Check load ===

  watch cat /proc/loadavg

  top
  
  sar -u 5


Results: Cpu(s): 11.4%us, 29.6%sy, 0.0%ni, 58.3%id, .7%wa, 0.0%hi, 0.0%si, 0.0%st

IF "wa" I/O wait is high, the load is most probably Disc bound (or RAM) 

IF "sy" System is high, it might be, **mdadm rund a RAID check**


=== Check Disc usage ===

  iotop -oP
  
  iostat 

=== Check, which processes are waiting to write to disc ===

This can really help!

  ps -e v


Check which processes are in R or D state:

  ps -eo s,user,cmd | grep ^[RD] | sort | uniq -c | sort -nbr | head -20


This outputs the state of each process. Specially D is interesting (but also R):

  * D    Marks a process in disk (or other short term, uninterruptible) wait.
  * R    Marks a runnable process


PSN - Linux Process Snapper is another nice tool:

Install from: https://tanelpoder.com/psnapper/


=== Links ===

  * https://www.linuxjournal.com/article/10688
  * https://www.tummy.com/articles/isolating-heavy-load/
  * https://tanelpoder.com/posts/high-system-load-low-cpu-utilization-on-linux/