• munin graphs
• monit service check & alarm system

“Munin is a networked resource monitoring tool that can help analyze resource trends and “what just happened to kill our performance?” problems”… http://munin-monitoring.org/

• “munin” is the central server - run as cronjob every 5 min
• “munin-node” is a client - run as service

Configuration:

http://munin-monitoring.org/wiki/munin-node-configure

Plugin base setup:

munin-node-configure --suggest   # test 
munin-node-configure --shell     # install symlinks

Test a plugin:

munin-run <plugin-name>

Test Server connection:

It should reply with the same hostname as setup in munin.conf. If not, edit munin-node.conf host_name on client

telnet <ip or localhost> <port>   # default port is 4949

Debug:

http://munin-monitoring.org/wiki/FAQ_no_graphs

it is possible to tunnel the master's requests over ssh, instead of opening another port: run this on the node:

ssh -L 4950:localhost:4949 -f -N -p22 -M -S /var/run/ssh_tunnel_xs.sock -o ExitOnForwardFailure=yes root@10.23.0.2

this tunnels munin port 4949 on the node to port 4950 on the master, using a socket to be able to stop the tunnel:

ssh -S /var/run/ssh_tunnel_xs.sock -O exit root@10.23.0.2

SSH tunnel init script to autostart on boot: sshtunnel.sh

autossh would be better to keep the connection, but this doesnt work:

autossh -M 4951 -N -R 4949:localhost:4950 -p22 root@10.23.0.2  

edit /etc/munin/munin.conf to access the node on localhost:

[nodebox]
  address 127.0.0.1
  use_node_name yes
  port 4950 

use ~/.ssh/authorized_keys to authenticate. protect the key by prepending these options:

from="10.23.0.1",command="/bin/false",no-pty,no-X11-forwarding,no-agent-forwarding,no-port-forwarding,permitopen="localhost:4949"
ssh-dss AAAAB3......

more info: http://munin-monitoring.org/wiki/faq#Q:HowcanIuseanSSHtunneltoconnecttoanode

ln -s /usr/share/munin/plugins/smart_ /etc/munin/plugins/smart_sda
ln -s /usr/share/munin/plugins/smart_ /etc/munin/plugins/smart_sdb

http://localhost/nginx_status required for nginx plugin

ln -s /usr/share/munin/plugins/nginx_* /etc/munin/plugins/

nginx vhost /sites-available/local:

server {
  listen   127.0.0.1:80;

  access_log off;
  server_name _; 
  server_name_in_redirect off;
  root  /home/local;

  # nginx_status configuration, need for Munin
  location /nginx_status {
      stub_status on;
      access_log   off;
      allow 127.0.0.1;
      deny all;
  }
  location / {
      return 301 http://<domain>$request_uri;
  }
}

required packages:

apt-get install  libio-all-lwp-perl 

/etc/munin/plugin-conf.d/munin-node:

[nginx*]
env.url http://127.0.0.1/nginx_status

mysql user required:

CREATE USER 'munin'@'localhost' IDENTIFIED BY 'PASSWD_HERE';
GRANT PROCESS, REPLICATION CLIENT ON *.* TO 'munin'@'localhost';

apt-get install libcache-{perl,cache-perl}

/etc/munin/plugin-conf.d/munin-node:

[mysql*]
env.mysqlconnection DBI:mysql:information_schema;host=127.0.0.1;port=3306
env.mysqluser munin
env.mysqlpassword PASSWD_HERE
env.cachenamespace munin_mysql_pri

https://github.com/MorbZ/munin-php-fpm

follow install howto

ln -s /usr/share/munin/plugins/http_loadtime /etc/munin/plugins/

/etc/munin/plugin-conf.d/munin-node:

[http_loadtime]
env.target https://<domain>/

nginx access by project

log analyser: https://github.com/munin-monitoring/contrib/blob/master/plugins/nginx/nginx_byprojects/byprojects_access

• needs logtail
• access.logs must be configured in /usr/share/munin/plugins/nginx_byprojects_access
• : doesn't work

Alerts:

https://www.linuxquestions.org/questions/linux-software-2/munin-changing-cpu-notify-threshold-606308/