• munin graphs
• monit service check & alarm system

“Munin is a networked resource monitoring tool that can help analyze resource trends and “what just happened to kill our performance?” problems”… http://munin-monitoring.org/

…tbd…

it is possible to tunnel the master's requests over ssh, instead of opening another port: run this on the node:

ssh -L 4950:localhost:4949 -f -N -p22 -M -S /var/run/ssh_tunnel_xs.sock -o ExitOnForwardFailure=yes root@10.23.0.2

this tunnels munin port 4949 on the node to port 4950 on the master, using a socket to be able to stop the tunnel:

ssh -S /var/run/ssh_tunnel_xs.sock -O exit root@10.23.0.2

autossh would be better to keep the connection, but this doesnt work:

autossh -M 4951 -N -R 4949:localhost:4950 -p22 root@10.23.0.2  

edit /etc/munin/munin.conf to access the node on localhost:

[nodebox]
  address 127.0.0.1
  use_node_name yes
  port 4950 

use ~/.ssh/authorized_keys to authenticate. protect the key by prepending these options:

from="10.23.0.1",command="/bin/false",no-pty,no-X11-forwarding,no-agent-forwarding,no-port-forwarding,permitopen="localhost:4949"
ssh-dss AAAAB3......

more info: http://munin-monitoring.org/wiki/faq#Q:HowcanIuseanSSHtunneltoconnecttoanode