High Availability with HAProxy and Keepalived in AWS

Posted by on

High Availability with HAProxy and Keepalived in AWS

Although AWS offers its own load balancer (ELB), my preference goes out to HAProxy, since I can configure it entirely the way I would like to. This only requires me to create my own failover system. Luckily we can use Keepalived for that. In this tutorial we will be creating 2 HAProxy EC2 instances. We will turn off the master instance and make sure the second instance takes over.

You can follow my article to set up a custom VPC in which you can test out this set up.

Prepare EC2 Instances

First you have to make sure you have 2 up and running EC2 instances.
We call our main EC2 instance Callisto and our secondary EC2 instance Ganymede.

I’ve used the Amazon Linux AMI images for both instances.

Make sure to also create a role for both with permissions to perform changes to EC2 instances.

Install HAProxy

You need to install HAProxy on both servers.

yum install haproxy

I used the following configuration file in /etc/haproxy/haproxy.cfg

global
    log         127.0.0.1 local0 
    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4000
    user        haproxy
    group       haproxy
    daemon
    stats socket /var/lib/haproxy/stats

defaults
    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option http-server-close
    option forwardfor       except 127.0.0.0/8
    option                  redispatch
    retries                 3
    timeout http-request    10s
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           10s
    maxconn                 3000

    errorfile 503 /etc/haproxy/errorfiles/503.http

frontend http-in
    mode http
    bind *:80 

    default_backend varnish

backend varnish
    mode http
    balance roundrobin

    option httpchk HEAD / HTTP/1.1
    server varnish 127.0.0.1:6081 check

The actual varnish backend does not exist. In your own environment, you would make sure of course that you have a proper backend configured. What is important for now is that we configured a 503 error file as an easy check to see if our failover works.

Make sure the file /etc/haproxy/errorfiles/503.http exists with the following content.

HTTP/1.0 503 Service Unavailable
Cache-Control: no-cache
Connection: close
Content-Type: text/html

<html>
  <head>
    <title>503 -  Service Unavailable</title>
  </head>
  <body>
    <div>
          <h2>This is {servername}.</h2>
    </div>
  </body>
</html>

To see that the failover works, make sure to change the text of {servername} to the name of the server. This way we can show on which server we are currently running.

Lets configure HAProxy to start on boot and to make sure it’s up and running.

chkconfig haproxy on
service haproxy start

If you access both HAProxy instances through your browser, you will see the error pages you configured.

Elastic IP

Our next step is to attach an Elastic IP to our main server (Callisto). After you have done so, you can configure it within your DNS so you have an URL to try out. When visiting the URL or IP, you should see your welcomes message.

Install keepalived

Next up, we have to install Keepalived on both servers.
Keepalived will keep track of which server is currently the master server and when a failover should occur.

yum install keepalived

Callisto is our main server. Therefor we tell keepalived that this is the master. Make sure that the config file /etc/keepalived/keepalived.conf looks like this.

vrrp_script check_haproxy
{
    script "pidof haproxy"
    interval 5
    fall 2
    rise 2
}

vrrp_instance VI_1
{
    debug 2
    interface eth0
    state MASTER
    virtual_router_id 1
    priority 110
    unicast_src_ip 10.0.1.131

    unicast_peer
    {
        10.0.1.42
    }

    track_script
    {
        check_haproxy
    }

    notify_master /etc/keepalived/failover.sh
}

Make sure to set the unicast_src_ip to the private IP of the current server. Set the unicat_peer to the private IP of the Ganymede server.

Ganymede is our secondary server. Therefor we tell Keepalived that this is the backup server. Make sure that the config file /etc/keepalived/keepalived.conf looks like this.

vrrp_script check_haproxy
{
    script "pidof haproxy"
    interval 5
    fall 2
    rise 2
}

vrrp_instance VI_1
{
    debug 2
    interface eth0
    state BACKUP
    virtual_router_id 1
    priority 100
    unicast_src_ip 10.0.1.42

    unicast_peer
    {
        10.0.1.131
    }

    track_script
    {
        check_haproxy
    }

    notify_master /etc/keepalived/failover.sh
}

Make sure to set the unicast_src_ip to the private IP of the current server. Set the unicat_peer to the private IP of the Callisto server.

Next, create the /etc/keepalived/failover.sh on both servers. Make sure that the instance_id has the instance ID of the other server.

#!/bin/bash

EIP=52.212.151.17
INSTANCE_ID=i-0bdd8a68eb573fd1a

/usr/bin/aws ec2 disassociate-address --public-ip $EIP
/usr/bin/aws ec2 associate-address --public-ip $EIP --instance-id $INSTANCE_ID

Make sure the file is executable.

chmod 700 master.sh

Lets configure Keepalived to start on boot and to make sure it’s up and running.

chkconfig keepalived on
service keepalived start

Test your setup

If you do a tail -f /var/log/messages you will see the following message appear at Callisto.

Keepalived_vrrp[1196]: VRRP_Instance(VI_1) Entering MASTER STATE

Whereas Ganymede has the following message.

Keepalived_vrrp[784]: VRRP_Instance(VI_1) Entering BACKUP STATE

If this is the case, you should be good to go. If you now call the Elastic IP in your browser, you would see the welcomes message from Callisto.

Now stop HAProxy on Callisto.

service haproxy stop

Reload your browser. The message of Ganymede should appear. You now created a failover for your main HAProxy instance.

Posted in AWS  · 

9 thoughts on “High Availability with HAProxy and Keepalived in AWS”

  1. Thanks for the article. I am trying to replicate this setup using docker. Just wonder if it is possible to run haproxy/keepalived in container and map from docker ip back to the aws instance private ip?

    Reply

    2. Vincent Fong, this setup is possible but pointless.
      Keepalived is L3/L4 balancer. It distribute IP traffic. So it makes sense only if you are using scheme when single HAProxy container exists on single docker host and you provide keepalived HA for hosts, not for containers.
      For containers in AWS it is better to use ALB/ELB/NLB as entry point for traffic – aws containers(tasks) has option to register in AWS Load Balancer target groups upon creation (and with random port, what is much better in case when many containers can run on single host)
      Also putting docker host in a public network is not generally recommended because of security concerns

      Reply

  2. Thank you for sharing, but I have a question to ask. When EIP is bound to master, master EC2 cannot be reconnected. How to execute the service haproxy stop command in master ec2?
    Look forward to your reply:-)

    Reply

  4. hello thanks for this solution
    but this is not right

    the file should Have the ID of the instance itself cuz only the master can execute the file
    Next, create the /etc/keepalived/failover.sh on both servers. Make sure that the instance_id has the instance ID of the other server.

    Reply

  5. my both machine has same output while execute the keepalived start which is
    Entering BACKUP STATE (init) then
    Entering MASTER STATE
    what is the solutions

    Reply

    1. Check that your priority is set to 110 on the primary server and 100 on the second server. If it is the same they will both try and be master. In which case the elastic IP would be on the last node that didn’t run the failover.sh

      Reply

Leave a Reply

Your email address will not be published. Required fields are marked *