High Availability with HAProxy and Keepalived in AWS


Posted by Peter Nijssen

High Availability with HAProxy and Keepalived in AWS

Although AWS offers its own load balancer (ELB), my preference goes out to HAProxy, since I can configure it entirely the way I would like to. This only requires me to create my own failover system. Luckily we can use Keepalived for that. In this tutorial we will be creating 2 HAProxy EC2 instances. We will turn off the master instance and make sure the second instance takes over.

Prepare EC2 Instances

First you have to make sure you have 2 up and running EC2 instances. We call our main EC2 instance Callisto and our secondary EC2 instance Ganymede.

I've used the Amazon Linux AMI images for both instances.

Make sure to also create a role for both with permissions to perform changes to EC2 instances.

Install HAProxy

You need to install HAProxy on both servers.

yum install haproxy

I used the following configuration file in /etc/haproxy/haproxy.cfg

global
    log         127.0.0.1 local0 
    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4000
    user        haproxy
    group       haproxy
    daemon
    stats socket /var/lib/haproxy/stats

defaults
    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option http-server-close
    option forwardfor       except 127.0.0.0/8
    option                  redispatch
    retries                 3
    timeout http-request    10s
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           10s
    maxconn                 3000

    errorfile 503 /etc/haproxy/errorfiles/503.http

frontend http-in
    mode http
    bind *:80 

    default_backend varnish

backend varnish
    mode http
    balance roundrobin

    option httpchk HEAD / HTTP/1.1
    server varnish 127.0.0.1:6081 check

The actual varnish backend does not exist. In your own environment, you would make sure of course that you have a proper backend configured. What is important for now is that we configured a 503 error file as an easy check to see if our failover works.

Make sure the file /etc/haproxy/errorfiles/503.http exists with the following content.

HTTP/1.0 503 Service Unavailable
Cache-Control: no-cache
Connection: close
Content-Type: text/html

<html>
  <head>
    <title>503 -  Service Unavailable</title>
  </head>
  <body>
    <div>
          <h2>This is {servername}.</h2>
    </div>
  </body>
</html>

To see that the failover works, make sure to change the text of {servername} to the name of the server. This way we can show on which server we are currently running.

Lets configure HAProxy to start on boot and to make sure it's up and running.

chkconfig haproxy on
service haproxy start

If you access both HAProxy instances through your browser, you will see the error pages you configured.

Elastic IP

Our next step is to attach an Elastic IP to our main server (Callisto). When you have done so, you can configure it within your DNS so you have an URL to try out. When visiting the URL or IP, you should see your welcomes message.

Install keepalived

Next up, we have to install Keepalived on both servers. Keepalived will keep track of which server is currently the master server and when a failover should occur.

yum install keepalived

Callisto is our main server. Therefor we tell keepalived that this is the master. Make sure that the config file /etc/keepalived/keepalived.conf looks like this.

vrrp_script check_haproxy
{
    script "pidof haproxy"
    interval 5
    fall 2
    rise 2
}

vrrp_instance VI_1
{
    debug 2
    interface eth0
    state MASTER
    virtual_router_id 1
    priority 110
    unicast_src_ip 10.0.1.131

    unicast_peer
    {
        10.0.1.42
    }

    track_script
    {
        check_haproxy
    }

    notify_master /etc/keepalived/failover.sh
}

Make sure to set the unicast_src_ip to the private IP of the current server. Set the unicat_peer to the private IP of the Ganymede server.

Ganymede is our secondary server. Therefor we tell Keepalived that this is the backup server. Make sure that the config file /etc/keepalived/keepalived.conf looks like this.

vrrp_script check_haproxy
{
    script "pidof haproxy"
    interval 5
    fall 2
    rise 2
}

vrrp_instance VI_1
{
    debug 2
    interface eth0
    state BACKUP
    virtual_router_id 1
    priority 100
    unicast_src_ip 10.0.1.42

    unicast_peer
    {
        10.0.1.131
    }

    track_script
    {
        check_haproxy
    }

    notify_master /etc/keepalived/failover.sh
}

Make sure to set the unicast_src_ip to the private IP of the current server. Set the unicat_peer to the private IP of the Callisto server.

Next, create the /etc/keepalived/failover.sh on both servers. Make sure that the instance_id has the instance ID of the other server.

#!/bin/bash

EIP=52.212.151.17
INSTANCE_ID=i-0bdd8a68eb573fd1a

/usr/bin/aws ec2 disassociate-address --public-ip $EIP
/usr/bin/aws ec2 associate-address --public-ip $EIP --instance-id $INSTANCE_ID

Make sure the file is executable.

chmod 700 master.sh

Lets configure Keepalived to start on boot and to make sure it's up and running.

chkconfig keepalived on
service keepalived start

Test your setup

If you do a tail -f /var/log/messages you will see the following message appear at Callisto.

Keepalived_vrrp[1196]: VRRP_Instance(VI_1) Entering MASTER STATE

Where Ganymede has the following message.

Keepalived_vrrp[784]: VRRP_Instance(VI_1) Entering BACKUP STATE

If this is the case, you should be good to go. If you now call the Elastic IP in your browser, you would see the welcomes message from Callisto.

Now stop HAProxy on Callisto.

service haproxy stop

Reload your browser. The message of Ganymede should appear. You now created a failover for your main HAProxy instance.


3 Comments

avatar
Nice article, I've added a link on the haproxy.org home page
avatar
Thanks!
avatar
How do you avoid split brain?