====== Monitoring ======

  * munin graphs
  * monit service check & alarm system


===== monit =====

syntax is a bit tricky, but it is a reliable system..

TBD


monit mailq via external script: https://lists.nongnu.org/archive/html/monit-general/2015-01/msg00015.html

===== munin =====

"Munin is a networked resource monitoring tool that can help analyze resource trends and "what just happened to kill our performance?" problems"... http://munin-monitoring.org/

  * "munin" is the central server - run as cronjob every 5 min
  * "munin-node" is a client - run as service


**Configuration:**

http://munin-monitoring.org/wiki/munin-node-configure

**Plugin base setup:**

  munin-node-configure --suggest   # test 
  munin-node-configure --shell     # install symlinks

**Test a plugin:**

  munin-run <plugin-name>

**Test Server connection:**

It should reply with the same hostname as setup in munin.conf. If not, edit munin-node.conf host_name on client

  telnet <ip or localhost> <port>   # default port is 4949


**Debug:**

http://munin-monitoring.org/wiki/FAQ_no_graphs


==== access munin-nodes via SSH tunnel ====

it is possible to tunnel the master's requests over ssh, instead of opening another port:
run this on the node:

  ssh -L 4950:localhost:4949 -f -N -p22 -M -S /var/run/ssh_tunnel_xs.sock -o ExitOnForwardFailure=yes root@10.23.0.2

this tunnels munin port 4949 on the node to port 4950 on the master, using a socket to be able to stop the tunnel:

  ssh -S /var/run/ssh_tunnel_xs.sock -O exit root@10.23.0.2

SSH tunnel init script to autostart on boot: {{:communitycloud:services:sshtunnel.sh|}}


\\
autossh would be better to keep the connection, but this doesnt work: FIXME
  autossh -M 4951 -N -R 4949:localhost:4950 -p22 root@10.23.0.2  


\\
edit **/etc/munin/munin.conf** to access the node on localhost:

  [nodebox]
    address 127.0.0.1
    use_node_name yes
    port 4950 


use **~/.ssh/authorized_keys** to authenticate. protect the key by prepending these options:

  from="10.23.0.1",command="/bin/false",no-pty,no-X11-forwarding,no-agent-forwarding,no-port-forwarding,permitopen="localhost:4949"
  ssh-dss AAAAB3......


more info: http://munin-monitoring.org/wiki/faq#Q:HowcanIuseanSSHtunneltoconnecttoanode


==== Plugins ====

=== smart ===

  ln -s /usr/share/munin/plugins/smart_ /etc/munin/plugins/smart_sda
  ln -s /usr/share/munin/plugins/smart_ /etc/munin/plugins/smart_sdb


=== nginx ===

http://localhost/nginx_status required for nginx plugin
  ln -s /usr/share/munin/plugins/nginx_* /etc/munin/plugins/

nginx vhost /sites-available/local:

  server {
    listen   127.0.0.1:80;

    access_log off;
    server_name _; 
    server_name_in_redirect off;
    root  /home/local;

    # nginx_status configuration, need for Munin
    location /nginx_status {
        stub_status on;
        access_log   off;
        allow 127.0.0.1;
        deny all;
    }
    location / {
        return 301 http://<domain>$request_uri;
    }
  }

required packages:

  apt-get install  libio-all-lwp-perl 

/etc/munin/plugin-conf.d/munin-node:

  [nginx*]
   env.url http://127.0.0.1/nginx_status


=== apache ===

needs the apache status page

config:
  [apache]
   env.url   http://127.0.0.1:%d/server-status?auto
   env.ports 80

packages:
  apt install libwww-perl

symlinks:
  ln -s /usr/share/munin/plugins/apache_* /etc/munin/plugins/



=== mysql / mariadb ===

mysql user required:
  CREATE USER 'munin'@'localhost' IDENTIFIED BY 'PASSWD_HERE';
  GRANT PROCESS, REPLICATION CLIENT ON *.* TO 'munin'@'localhost';

  apt-get install libcache-{perl,cache-perl} libdbd-mysql-perl


/etc/munin/plugin-conf.d/munin-node:

  [mysql*]
   env.mysqlconnection DBI:mysql:information_schema;host=127.0.0.1;port=3306
   env.mysqluser munin
   env.mysqlpassword PASSWD_HERE
   env.cachenamespace munin_mysql_pri
  
   # for myisam / queries..:
   env.mysqlopts -h localhost -u munin --password=PASSWD_HERE

enable replication graph:
  ln -s /usr/share/munin/plugins/mysql_ /etc/munin/plugins/mysql_replication

bugfix: https://github.com/munin-monitoring/munin/pull/109/

=== php-fpm ===

https://github.com/MorbZ/munin-php-fpm

follow install howto

=== http wget website loadtime ===
  ln -s /usr/share/munin/plugins/http_loadtime /etc/munin/plugins/

/etc/munin/plugin-conf.d/munin-node:

  [http_loadtime]
  env.target https://<domain>/

=== processes ===

just name the symlink like the program you want to monitor (e.g. rsync):

  ln -s /usr/share/munin/plugins/ps_ /etc/munin/plugins/ps_rsync



----

=== todo ===

very good list and setup of plugins: http://kevin.deldycke.com/2011/06/munin-monitor-debian-squeeze-server/


nginx access by project

log analyser:  https://github.com/munin-monitoring/contrib/blob/master/plugins/nginx/nginx_byprojects/byprojects_access
  * needs logtail
  * access.logs must be configured in /usr/share/munin/plugins/nginx_byprojects_access
  * FIXME: doesn't work


Alerts:

Do not work..for me.. yet..:

  * https://www.linuxquestions.org/questions/linux-software-2/munin-changing-cpu-notify-threshold-606308/
  * http://blog-en.openalfa.com/how-to-configure-alerts-in-munin