High availability through Keepalive

Keepalive is a great high availability system that allows us to automatically migrate IPs between different servers, enabling us to have a highly available system with minimal downtime.

In this example, we will use Keepalive to maintain several shared IPs (VIPs) for the HaProxy process.

Keepalive operates on the VRRP protocol at level 2, which means that the devices must be on the same network segment.

Devices with the same configured PASS are monitored between each other. If the VIPs go down, they are automatically configured on the indicated interface.

NOTE: Several Keepalives can be on the same LAN, but each one must have different router_id, virtual_router_id, and vrrp_instance.

emerge -av keepalived

On ha10:

vi /etc/keepalived/keepalived.conf

! Configuration File for keepalived

global_defs {
   notification_email {
     sys@alfaexploit.com
   }
   notification_email_from ha10@rack
   smtp_server localhost
   smtp_connect_timeout 30
   router_id LVS_MAIN_HA
}
vrrp_script chk_haproxy {
    script "killall -0 haproxy"
    interval 2 # check cada 2 segundos

    weight 2 # anyade 2 puntos de prioridad si OK

}

vrrp_instance VI_HA_1 {
    state MASTER
    interface eth1
    virtual_router_id 10
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass XXXXXXXXX
    }
    virtual_ipaddress {
        AAA.BBB.CCC.DDD/MASK dev INTERFAZ
        AAA.BBB.CCC.DDD/MASK dev INTERFAZ
        AAA.BBB.CCC.DDD/MASK dev INTERFAZ
        AAA.BBB.CCC.DDD/MASK dev INTERFAZ
        AAA.BBB.CCC.DDD/MASK dev INTERFAZ
        AAA.BBB.CCC.DDD/MASK dev INTERFAZ
        AAA.BBB.CCC.DDD/MASK dev INTERFAZ
        AAA.BBB.CCC.DDD/MASK dev INTERFAZ
        AAA.BBB.CCC.DDD/MASK dev INTERFAZ
        AAA.BBB.CCC.DDD/MASK dev INTERFAZ
        AAA.BBB.CCC.DDD/MASK dev INTERFAZ
        AAA.BBB.CCC.DDD/MASK dev INTERFAZ
        AAA.BBB.CCC.DDD/MASK dev INTERFAZ
        AAA.BBB.CCC.DDD/MASK dev INTERFAZ
    }

    track_script {
        chk_haproxy
    }
}

On ha11:

vi /etc/keepalived/keepalived.conf

! Configuration File for keepalived

global_defs {
   notification_email {
     sys@alfaexploit.com
   }
   notification_email_from ha11@rack
   smtp_server localhost
   smtp_connect_timeout 30
   router_id LVS_MAIN_HA
}
vrrp_script chk_haproxy {
    script "killall -0 haproxy"
    interval 2
    weight 2
}
vrrp_instance VI_HA_1 {
    state BACKUP
    interface eth1
    virtual_router_id 10
    priority 99
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass XXXXXXXXX
    }

    virtual_ipaddress {

        AAA.BBB.CCC.DDD/MASK dev INTERFAZ
        AAA.BBB.CCC.DDD/MASK dev INTERFAZ
        AAA.BBB.CCC.DDD/MASK dev INTERFAZ
        AAA.BBB.CCC.DDD/MASK dev INTERFAZ
        AAA.BBB.CCC.DDD/MASK dev INTERFAZ
        AAA.BBB.CCC.DDD/MASK dev INTERFAZ
        AAA.BBB.CCC.DDD/MASK dev INTERFAZ
        AAA.BBB.CCC.DDD/MASK dev INTERFAZ
        AAA.BBB.CCC.DDD/MASK dev INTERFAZ
        AAA.BBB.CCC.DDD/MASK dev INTERFAZ
        AAA.BBB.CCC.DDD/MASK dev INTERFAZ
        AAA.BBB.CCC.DDD/MASK dev INTERFAZ
        AAA.BBB.CCC.DDD/MASK dev INTERFAZ
        AAA.BBB.CCC.DDD/MASK dev INTERFAZ
    }

    track_script {
        chk_haproxy
    }
}

We have configured priorities 100 and 99, and weights 2 (in the check script). While it is active:

ha10: 100+2=102 – INACTIVE–> 100
ha11: 99+2=101 – INACTIVE–> 99

The kill -0 command checks that the ha service is active. When this check fails, the IPs will be migrated automatically.

WARNING:
We must take into account several aspects for all of this to work automatically. We must leave the service configured with the final IPs in the ha, and to bind some IPs that it does not have in the backup, we must do some tricks.

vi /etc/sysctl.conf

net.ipv4.ip_forward = 1
net.ipv4.conf.default.rp_filter = 0
net.ipv4.conf.all.rp_filter = 0
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.default.accept_source_route = 0
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.default.accept_redirects = 0
net.ipv4.conf.all.secure_redirects = 0
net.ipv4.conf.default.secure_redirects = 0
net.ipv4.ip_nonlocal_bind=1

sysctl -p

Start keepalive and add it to the startup:

/etc/init.d/keepalived start
rc-update add keepalived default

NOTE: VRRP has a limit of 20 vips per instance, if more are required, a second instance must be configured.

There are networks in which the router’s ARP table is not refreshed correctly, to force it:

arping -s VIP -I eth0 GW_IP

This can be automated with:

vrrp_instance VI_HA_1 {
    state BACKUP
    interface eth0
    virtual_router_id 10
    priority 100
    notify_master /root/keepMASTER.sh

vi /root/keepMASTER.sh

#! /bin/bash
VIP="A.B.C.D"
IF="eth1"
IPROUTER="E.F.G.H"
DATE=$(date)
echo -e "NEW ROLE: MASTER -- ReARPing Router ARP table: $DATE" >> /tmp/keepalived.log
arping -s $VIP -I $IF -c 5 $IPROUTER

This article is very interesting and I hope you get the most out of it ;)