Installing Zabbix Server on Centos6

Zabbix as stated on official website is an “enterprise-level software designed for monitoring availability and performance of IT infrastructure components”.
It’s handy if you need to monitor a handful of servers and supports most major linux distributions, Windows, VMware and Java Applications. It’s also fairly easy to setup and understand with a short learning curve offering many built in “scenarios” and fancy graphs. I personally prefer it over other similar solutions like Nagios.
So let’s go through installing and configuring. There are two options, you can either use the official guide where you’ll have to go through all the steps and manually setup things or use a fancy ansible script and do things automagically! We’ll use the ansible script to make our lives easier.
Ansible is an automation tool that is able to deploy and configure software on multiple hosts simultaneously. Comparing it to other similar tools (chef, puppet, etc.) i find it much easier to read, write and maintain, has a big set of premade scripts that are expanding all the time at ansible galaxy and is agentless. All you need is ssh access to your machines. So let’s go through installing ansible and running our playbook.
You can install ansible on any machine you’d like and we’ll run the playbooks from there. On Fedora you can install it through the official repos with

sudo yum install ansible -y

We’ll use a premade playbook that i’ve found on ansible galaxy that installs zabbix server and agent on a Centos 6 system with MySQL and apache. First fetch the playbook.

sudo ansible-galaxy install akamine.c6-zabbix

Then edit the default configuration here /etc/ansible/roles/akamine.c6-zabbix/defaults/main.yml and follow the below steps.

 mkdir zabbix-ansible
cd zabbix-ansible
vi hosts
vi zabbix.yml
ansible-playbook -i hosts zabbix.yml

On the hosts file you’ll put the hosts you wish the playbook to run.
An example is as follows

[zabbix-server]
my.leet.host

And on zabbix.yml

---
- hosts: all
  roles:
  - akamine.c6-zabbix

When playing the book you should see it output what is doing similar to this:

akis@localhost ansible$ ansible-playbook -i zabbix.host zabbix.yml 

PLAY [all] ******************************************************************** 

GATHERING FACTS *************************************************************** 
ok: [my.leet.host]

TASK: [akamine.c6-zabbix | install libselinux-python] ************************* 
changed: [my.leet.host]

TASK: [akamine.c6-zabbix | install MySQL-python] ****************************** 
changed: [my.leet.host]

TASK: [akamine.c6-zabbix | change selinux mode to permissive] ***************** 
changed: [my.leet.host]

TASK: [akamine.c6-zabbix | disable iptables] ********************************** 
changed: [my.leet.host]

TASK: [akamine.c6-zabbix | install ntp] *************************************** 
changed: [my.leet.host]

TASK: [akamine.c6-zabbix | Start and enable ntp service] ********************** 
changed: [my.leet.host]

TASK: [akamine.c6-zabbix | install epel repo rpm key] ************************* 
changed: [my.leet.host]

TASK: [akamine.c6-zabbix | setup epel repo] *********************************** 
changed: [my.leet.host]

TASK: [akamine.c6-zabbix | install htop] ************************************** 
changed: [my.leet.host]

TASK: [akamine.c6-zabbix | install tmux] ************************************** 
changed: [my.leet.host]

TASK: [akamine.c6-zabbix | install mysql server] ****************************** 
changed: [my.leet.host]

TASK: [akamine.c6-zabbix | enable mysql server] ******************************* 
changed: [my.leet.host]

TASK: [akamine.c6-zabbix | set innodb_buffer_size in /etc/my.cnf] ************* 
changed: [my.leet.host]

TASK: [akamine.c6-zabbix | enable event_scheduler in /etc/my.cnf] ************* 
changed: [my.leet.host]

TASK: [akamine.c6-zabbix | install zabbix rpm key] **************************** 
changed: [my.leet.host]

TASK: [akamine.c6-zabbix | setup zabbix repo] ********************************* 
changed: [my.leet.host]

TASK: [akamine.c6-zabbix | install zabbix-server-mysql] *********************** 
changed: [my.leet.host]

TASK: [akamine.c6-zabbix | install zabbix-web-mysql] ************************** 
changed: [my.leet.host]

TASK: [akamine.c6-zabbix | install zabbix-agent] ****************************** 
changed: [my.leet.host]

TASK: [akamine.c6-zabbix | create database zabbix] **************************** 
changed: [my.leet.host]

TASK: [akamine.c6-zabbix | ensure mysqld is running] ************************** 
ok: [my.leet.host]

TASK: [akamine.c6-zabbix | initialize zabbix database] ************************ 
changed: [my.leet.host] => (item=schema.sql)
changed: [my.leet.host] => (item=images.sql)
changed: [my.leet.host] => (item=data.sql)

TASK: [akamine.c6-zabbix | copy sql scripts for database partitioning] ******** 
changed: [my.leet.host] => (item=alter_table_indexes.sql)
changed: [my.leet.host] => (item=stored_procedures.sql)
changed: [my.leet.host] => (item=event_schedule.sql)

TASK: [akamine.c6-zabbix | alter table indexes] ******************************* 
changed: [my.leet.host]

TASK: [akamine.c6-zabbix | add stored procedures for partitioning] ************ 
changed: [my.leet.host]

TASK: [akamine.c6-zabbix | add partitioning event to schedule] **************** 
changed: [my.leet.host]

TASK: [akamine.c6-zabbix | create mysql user] ********************************* 
changed: [my.leet.host]

TASK: [akamine.c6-zabbix | modify /etc/zabbix/zabbix_server.conf] ************* 
changed: [my.leet.host]

TASK: [akamine.c6-zabbix | modify /etc/httpd/conf.d/zabbix.conf] ************** 
changed: [my.leet.host]

TASK: [akamine.c6-zabbix | add e-mail sending script to /usr/lib/zabbix/alertscripts] *** 
changed: [my.leet.host]

TASK: [akamine.c6-zabbix | enable httpd] ************************************** 
changed: [my.leet.host]

TASK: [akamine.c6-zabbix | enable zabbix server] ****************************** 
changed: [my.leet.host]

TASK: [akamine.c6-zabbix | enable and start zabbix agent] ********************* 
changed: [my.leet.host]

NOTIFIED: [akamine.c6-zabbix | restart zabbix server] ************************* 
changed: [my.leet.host]

NOTIFIED: [akamine.c6-zabbix | restart httpd] ********************************* 
changed: [my.leet.host]

NOTIFIED: [akamine.c6-zabbix | restart mysqld] ******************************** 
changed: [my.leet.host]

PLAY RECAP ******************************************************************** 
my.leet.host    : ok=37   changed=35   unreachable=0    failed=0

Next all you need to do is login to your new zabbix server instance on http://my.leet.host/zabbix with username Admin and pass zabbix.
Will go through configuring the server and installing agents in a later post.
Happy playing around with zabbix!

PS: Some quick notes..
The ansible playbook installs mysql-server with default options, meaning no root password.
You can go further editing the playbook adding these to zabbix-database-init.yml

- name: Update mysql root password for all root accounts
  mysql_user: name=root host={{ item }} password=huhu
  with_items: [ '{{ ansible_hostname }}', '127.0.0.1', '::1', 'localhost' ]

- name: Delete anonymous mysql user
  mysql_user: user="" host={{ item }} state=absent
  with_items: [ '{{ ansible_hostname }}', 'localhost' ]

- name: Remove test database
  mysql_db: db=test state=absent

This will setup a password for root account and remove test database.
I’ll probably do a fork of the repo and add some bits and pieces to it and update this post but for the time being this should do.

CentOS

Vmware ESXi NAT Networking

The case: You have multiple virtual machines that you don’t want to expose directly to the internet and you prefer routing them through a single virtual machine. This is also handy if you are short on IPs. Our case is OVH specific and it will give you a guide on how to set things up on OVH weird network topology. Reason why we use ipcop instead of pfsense is because pfsense doesn’t support a 32bit net mask. No matter what i’ve tried i could get it to work with a 32 bit netmask and gateway on separate network. So IpCop was the next best thing and it does the job quite well.

ESXi Configuration:

Login to your Vsphere client and go to Configuration -> Networking.

On Network Access select “Create a vSphere standard switch” without select any vmnic .

Proceed to Connection Settings and name your new nic under “Network Label” . “Vlan ID” select None.

Proceed on creating a new VM including both NICs the one that you just created and the one that connects you to the outside world.

Then move on with standard installation of IPcop. Keep in mind that your RED network will be the one connecting outside and the GREEN one will be your internal nat network. If you are not sure of the network settings while installing proceed and will edit network config later on anyways.

After installation login to the machine and now let’s edit the files needed for proper networking under OVH routers.

vi /var/ipcop/ethernet/settings

DEFAULT_GATEWAY=Dedicated Server IP but end in .254
DNS2=8.8.4.4
DNS1=8.8.8.8
RED_1_NETADDRESS=IP Failover
RED_1_NETMASK=255.255.255.255
RED_1_ADDRESS=IP Failover
RED_1_BROADCAST=IP Failover
GREEN_1_NETADDRESS=192.168.1.0
GREEN_1_NETMASK=255.255.255.0
GREEN_1_ADDRESS=192.168.1.1
GREEN_1_MAC=Your Virtual Mac Address
GREEN_1_DRIVER=pcnet32
GREEN_1_OPTIONS=
GREEN_1_DEV=lan-1
RED_1_MAC=Your Virtual Mac Address
RED_1_DRIVER=pcnet32
RED_1_OPTIONS=
RED_1_DEV=wan-1
RED_1_TYPE=STATIC
ORANGE_COUNT=0
BLUE_COUNT=0
RED_COUNT=1
GREEN_COUNT=1

vi /etc/rc.d/rc.event.local

#!/bin/sh
route add Dedicated Server IP but end in .254 dev wan-1
route add default gw Dedicated Server IP but end in .254

Reboot and you are done! You can now login to the web interface under https://192.168.1.1:8443

Bash

Bash: Script to monitor hard disk health and send an e-mail on error.

Here is a simple script to test for reallocated sector count and report through mail if it finds any.
Reallocated sectors means your disk has sectors where it can’t read/write thus it uses a reserved space to reallocate those sectors.
In general reallocated sectors are bad news for your disk and you should consider a backup and send to warranty.
The script reads the smart value and reports. It should be self explanatory to use it just type ./scriptname.sh nameofthehost. Afterwards set a cron to run daily.
Credits to Thorsten Rhau for the base script.

#!/bin/bash
# This script checks the health of disks

#Define a friendly name for this machine
mname="$1"

#Location to temporary store the error log
logloc="/root/scripts"
#Set default for not sendig mails
sendm="0"

# Disks to check
disks="/dev/sda
/dev/sdb"

# Setting up path
PATH="$PATH:/usr/bin:/usr/sbin"

# variable containing all needed commands
needed_commands="smartctl awk mail"

# Checking if all needed programs are available on system
for command in $needed_commands
do
  if ! hash "$command" > /dev/null 2>&1
  then
    echo "$command not found on system" 1>&2
    exit 1
  fi
done

# Checking disk
for disk in $disks
do
  # Creating a array with results
  declare -a status=(`smartctl -a -d ata $disk | awk '/Reallocated_Sector_Ct/ || /Seek_Error_Rate/ { print $2" "$NF }'`)
  # Checking that we do not have any Reallocated Sectors
  if [ "${status[1]}" -ne 0 ]
  then
    echo "$mname Warning: Disk $disk has errors! ${status[0]} ${status[1]} ${status[2]} ${status[3]}. Following complete smartctl output." >> diskerror.log
    smartctl -a -d ata $disk >> $logloc/diskerror.log
    failed=("${failed[@]}" "$disk")
    sendm="1"
  fi
done

#Send an e-mail if needed containing the failed diks (fdisks) info.
if [ $sendm == 1 ]; then
  fdisks=${failed[@]}
  mail -s "$mname Disks $fdisks are about to fail." user@localhost < $logloc/diskerror.log
  rm -rf $logloc/diskerror.log
fi
Bash

Bash: A very basic folder and MySQL bash backup script.

Here is a basic bash script to backup a folder and a MySQL database.

#!/bin/bash
#Backup script for www and mysql.

DATE=`date +%-Y%-m%-d`
backup_path="/path/to/where/iwant/mybackup/stored/"
backup_source="/path/of/myfolder/iwant/to/backup/"
user="db_username"
password="db_password"
host="db_host"
db_name="db_name"


# The function to actually do the backup.
function dobackup {
	echo "Starting backup for" $user
	echo "Backing up" $backup_source
	tar -cvzPf $backup_path$DATE-$user.tar.gz $backup_source
	echo "Backing up MySQL database.."
	mysqldump --user=$user --password=$password --host=$host $db_name > $backup_path/$db_name-$date.sql
	gzip $backup_path/$db_name-$date.sql
	echo "Backup done"
}

# Script should be run with arguement dobackup if we want to backup or delete if we want to delete todays backup.
if [ $1 = "dobackup" ] ; then
	dobackup
elif [ $1 = "delete" ] ; then
	rm -f $backup_path$DATE-$user.tar.gz
  	echo "Deleted old backup"
fi

Then we crontab this accordingly.

Alternatively you could add something like this to delete files older than 2 days on the backup path.

#Delete files older than 2 days
find $backup_path/* -mtime +2 -exec rm {} \;
Virtualbox
BitBucket

Migrating Subversion (SVN) to Git

We decided to move from a local subversion we were using to git using Bitbucket. Why moving from svn to git is a matter of taste imho. Whatever suits you best, there would definitely be pros and cons for each. We did it cause the programmers seemed to like working with git and we didn’t want to maintain an svn ourselves.  Bitbucket vs Github is all about the cost, in the end they are both git. In our case Bitbucket was cheaper since we are a small group of developers with large amount of repos, so the pricing of Bitbucket (per developer) is just right for us.

Bitbucket offers migration through their webpage but that was no option for us because we needed our commit history intact. So we were off for alternatives. Also our svn was pretty old so we had to do some extra steps before migrating. So let’s go through them.

Step 1

Install svn to a local machine.

sudo yum install mod_dav_svn subversion

For the rest you can follow this guide up to the point of creating a testrepo. Instead of creating testrepo we are going to take our own repos from the remote svn and put them in /var/www/svn.

Step 2
On the remote server.

tar -zcvf svnbackup.tar.gz /var/www/svn

Then get this to your local svn and extract accordingly.

Step 3

Give proper user rights

chown -R apache.apache *
chcon -R -t httpd_sys_rw_content_t /var/www/svn/*

And then goto your backuped svn folders and perform and svnadmin upgrade like this

 svnadmin upgrade /var/www/svn/myproject 

Step 4

Get a list of the authors so our commits get linked to them. Using this trick.

svn log --quiet http://localhost/svn/myproject \
| grep '^r' | awk '{print $3}' | sort | uniq > authors.txt

Step 5

Now the rest is simple using this guide.

git svn clone http://localhost/svn/svn.myrepo/ -A authors.txt my_repo
git init --bare my_bare_repo
cd my_bare_repo
git symbolic-ref HEAD refs/heads/trunk
cd ../my_repo
git remote add bare ../my_bare_repo
git config remote.bare.push 'refs/remotes/*:refs/heads/*'
git push bare
cd ../my_bare_repo
git branch -m git-svn master
git remote add origin https://me@bitbucket.org/me/myrepo.git
git push origin master

Bumps and how you can manage them!

    1. After git svn clone you end up with this error
fatal: refs/remotes/trunk: not a valid SHA1
update-ref HEAD refs/remotes/trunk: command returned error: 128

In most cases the below should do the job. Else do a git branch -a see your remotes and set master to follow one of them.

 git update-ref refs/heads/master refs/remotes/git-svn