Monitoring

  • munin graphs
  • monit service check & alarm system

monit

syntax is a bit tricky, but it is a reliable system..

TBD

monit mailq via external script: https://lists.nongnu.org/archive/html/monit-general/2015-01/msg00015.html

munin

“Munin is a networked resource monitoring tool that can help analyze resource trends and “what just happened to kill our performance?” problems”… http://munin-monitoring.org/

  • “munin” is the central server - run as cronjob every 5 min
  • “munin-node” is a client - run as service

Configuration:

http://munin-monitoring.org/wiki/munin-node-configure

Plugin base setup:

munin-node-configure --suggest   # test 
munin-node-configure --shell     # install symlinks

Test a plugin:

munin-run <plugin-name>

Test Server connection:

It should reply with the same hostname as setup in munin.conf. If not, edit munin-node.conf host_name on client

telnet <ip or localhost> <port>   # default port is 4949

Debug:

http://munin-monitoring.org/wiki/FAQ_no_graphs

access munin-nodes via SSH tunnel

it is possible to tunnel the master's requests over ssh, instead of opening another port: run this on the node:

ssh -L 4950:localhost:4949 -f -N -p22 -M -S /var/run/ssh_tunnel_xs.sock -o ExitOnForwardFailure=yes root@10.23.0.2

this tunnels munin port 4949 on the node to port 4950 on the master, using a socket to be able to stop the tunnel:

ssh -S /var/run/ssh_tunnel_xs.sock -O exit root@10.23.0.2

SSH tunnel init script to autostart on boot: sshtunnel.sh


autossh would be better to keep the connection, but this doesnt work: FIXME

autossh -M 4951 -N -R 4949:localhost:4950 -p22 root@10.23.0.2  


edit /etc/munin/munin.conf to access the node on localhost:

[nodebox]
  address 127.0.0.1
  use_node_name yes
  port 4950 

use ~/.ssh/authorized_keys to authenticate. protect the key by prepending these options:

from="10.23.0.1",command="/bin/false",no-pty,no-X11-forwarding,no-agent-forwarding,no-port-forwarding,permitopen="localhost:4949"
ssh-dss AAAAB3......

more info: http://munin-monitoring.org/wiki/faq#Q:HowcanIuseanSSHtunneltoconnecttoanode

Plugins

smart

ln -s /usr/share/munin/plugins/smart_ /etc/munin/plugins/smart_sda
ln -s /usr/share/munin/plugins/smart_ /etc/munin/plugins/smart_sdb

nginx

http://localhost/nginx_status required for nginx plugin

ln -s /usr/share/munin/plugins/nginx_* /etc/munin/plugins/

nginx vhost /sites-available/local:

server {
  listen   127.0.0.1:80;
  access_log off;
  server_name _; 
  server_name_in_redirect off;
  root  /home/local;
  # nginx_status configuration, need for Munin
  location /nginx_status {
      stub_status on;
      access_log   off;
      allow 127.0.0.1;
      deny all;
  }
  location / {
      return 301 http://<domain>$request_uri;
  }
}

required packages:

apt-get install  libio-all-lwp-perl 

/etc/munin/plugin-conf.d/munin-node:

[nginx*]
 env.url http://127.0.0.1/nginx_status

apache

needs the apache status page

config:

[apache]
 env.url   http://127.0.0.1:%d/server-status?auto
 env.ports 80

packages:

apt install libwww-perl

symlinks:

ln -s /usr/share/munin/plugins/apache_* /etc/munin/plugins/

mysql / mariadb

mysql user required:

CREATE USER 'munin'@'localhost' IDENTIFIED BY 'PASSWD_HERE';
GRANT PROCESS, REPLICATION CLIENT ON *.* TO 'munin'@'localhost';
apt-get install libcache-{perl,cache-perl} libdbd-mysql-perl

/etc/munin/plugin-conf.d/munin-node:

[mysql*]
 env.mysqlconnection DBI:mysql:information_schema;host=127.0.0.1;port=3306
 env.mysqluser munin
 env.mysqlpassword PASSWD_HERE
 env.cachenamespace munin_mysql_pri

 # for myisam / queries..:
 env.mysqlopts -h localhost -u munin --password=PASSWD_HERE

enable replication graph:

ln -s /usr/share/munin/plugins/mysql_ /etc/munin/plugins/mysql_replication

bugfix: https://github.com/munin-monitoring/munin/pull/109/

php-fpm

http wget website loadtime

ln -s /usr/share/munin/plugins/http_loadtime /etc/munin/plugins/

/etc/munin/plugin-conf.d/munin-node:

[http_loadtime]
env.target https://<domain>/

processes

just name the symlink like the program you want to monitor (e.g. rsync):

ln -s /usr/share/munin/plugins/ps_ /etc/munin/plugins/ps_rsync

todo

very good list and setup of plugins: http://kevin.deldycke.com/2011/06/munin-monitor-debian-squeeze-server/

nginx access by project

log analyser: https://github.com/munin-monitoring/contrib/blob/master/plugins/nginx/nginx_byprojects/byprojects_access

  • needs logtail
  • access.logs must be configured in /usr/share/munin/plugins/nginx_byprojects_access
  • FIXME: doesn't work

Alerts:

Do not work..for me.. yet..:

 
Back to top
communitycloud/services/monitoring.txt · Last modified: 2017/09/23 04:09 by tkilla