How to create a highly available web hosting platform using Floating IPs

In this Leaseweb Labs post, we’re going step-by-step to a proof of concept of a (very basic) highly available web hosting platform. Using Floating IPs and keepalived, we’ll create an active/standby setup on two different dedicated servers, with automatic failover through the Leaseweb API, so your application will never be down. We’ll use 2 dedicated servers and 1 Floating IP address from Leaseweb to make this happen.

What are Floating IPs?

Floating IPs are a kind of virtual IP address that can be dynamically routed to any server in the same network. Some hosting providers may also call this Elastic IPs or Virtual IP’s.

Multiple servers can own the same Floating IP address, but it can only be active on one server at any given time.

Floating IPs can be used to implement features such as:

  • Failover in a high-availability cluster
  • Zero-downtime Continuous Deployment

Using Floating IPs

Using Floating IPs is quite simple, with Leaseweb, you can order them through the customer portal and set them up on your server as an additional IP address. But the real power lies in automation. By using the Leaseweb API, it’s possible to use any script or even some 3rd party software to automatically control Floating IPs.

When paired with free software such as keepalived, which can detect when a server is down and take action accordingly, it becomes possible to create a fully automated highly available platform for any application.

Step one: Set up the servers and Floating IPs

First, let’s set up the two servers with a simple HTTP web server and use a Floating IP address to access the website of either one server.

  • Server A (Leaseweb Server Id 20483) has IP address 212.32.230.75 and is pre-installed with CentOS 7
  • Server B (Leaseweb Server Id 37089) has IP address 212.32.230.66 and is pre-installed with Ubuntu 18.04
  • 89.149.192.0 is the Floating IP address

Setting up the Floating IP address in the Customer Portal

If you don’t have a Floating IP yet, then from the Floating IPs page in the Leaseweb Customer Portal click the  button to order Floating IPs. Once delivered, you will see an entry like this:

Click on the range to open its detail page:

Here it is possible to set up a relationship between a Floating IP and an Anchor IP. Leaseweb calls this a “Floating IP Definition”, and can be done with the  button.

Let’s create a new definition to link Floating IP 89.149.192.0 to the Anchor IP 212.32.230.75 of server A:

Once saved, there will be one Floating IP Definition visible:

Setting up the Floating IP address and a demonstration webpage on the servers

On a server, a Floating IP can be set up as any other additional IP address. A gateway address is not necessary, and the subnet mask is always 255.255.255.255, or /32 in CIDR notation.

To add an additional IP address to an interface in Linux without making the change persistent, we can simply use the
ip -4 address show
command to show which device the main IP address is configured on, and then do
ip address add <Floating IP address>/32 dev <Device>
to add the floating IP to the same device.

We also install a HTTP server and create a simple demonstration webpage:

# Check which device we need to add then IP address to
ip -4 address show
ip address add 89.149.192.0/32 dev eno1

# The Floating IP address should now be visible on the device
ip -4 address show

# Install a web server and create a basic default webpage
yum install -y httpd
systemctl start httpd
cat <<EOF > /var/www/html/index.html
<!DOCTYPE html>
<html>
<head><title>This is test server A</title></head>
<body><h1>This is test server A</h1></body>
</html>
EOF

Result:

tim@laptop:~$ ssh root@20483.lsw
[root@servera ~]# ip -4 address show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    inet 212.32.230.75/26 brd 212.32.230.127 scope global eno1
       valid_lft forever preferred_lft forever

[root@servera ~]# ip address add 89.149.192.0/32 dev eno1

[root@servera ~]# ip -4 address show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    inet 212.32.230.75/26 brd 212.32.230.127 scope global eno1
       valid_lft forever preferred_lft forever
    inet 89.149.192.0/32 scope global eno1
       valid_lft forever preferred_lft forever

[root@servera ~]# yum install -y httpd

[...]

[root@servera ~]# systemctl start httpd

[root@servera ~]# cat <<EOF > /var/www/html/index.html
> <!DOCTYPE html>
> <html>
> <head><title>This is test server A</title></head>
> <body><h1>This is test server A</h1></body>
> </html>
> EOF

[root@servera ~]#

(note: ssh root@20483.lsw is a neat little trick explained here: https://gist.github.com/timwb/1f95737d54563aedd7c97d5e671667cc)

You should now already be able to ping the Floating IP address, and opening it in a browser loads the demo webpage:

Next, add the same Floating IP address to server B, install a HTTP web server and create a simple demo webpage:

# Check which device we need to add the IP address to
ip -4 address show
ip address add 89.149.192.0/32 dev enp32s0

# The Floating IP address should now be visible on the device
ip -4 address show

# Install a web server and create a basic default webpage
apt install -y nginx
cat <<EOF > /var/www/html/index.html
<!DOCTYPE html>
<html>
<head><title>This is test server B</title></head>
<body><h1>This is test server B</h1></body>
</html>
EOF

Result:

tim@laptop:~$ ssh root@37089.lsw
root@serverb:~# ip -4 address show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: enp32s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    inet 212.32.230.66/26 brd 212.32.230.127 scope global enp32s0
       valid_lft forever preferred_lft forever
3: enp34s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    inet 10.32.18.208/27 brd 10.32.18.223 scope global enp34s0
       valid_lft forever preferred_lft forever

root@serverb:~# ip address add 89.149.192.0/32 dev enp32s0

root@serverb:~# ip -4 address show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: enp32s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    inet 212.32.230.66/26 brd 212.32.230.127 scope global enp32s0
       valid_lft forever preferred_lft forever
    inet 89.149.192.0/32 scope global enp32s0
       valid_lft forever preferred_lft forever
3: enp34s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    inet 10.32.18.208/27 brd 10.32.18.223 scope global enp34s0
       valid_lft forever preferred_lft forever

root@serverb:~# apt install -y nginx

[...]

root@serverb:~# cat <<EOF > /var/www/html/index.html
> <!DOCTYPE html>
> <html>
> <head><title>This is test server B</title></head>
> <body><h1>This is test server B</h1></body>
> </html>
> EOF

root@serverb:~#

FLIP’ing a Floating IP

Initially, we’ve setup Floating IP 89.149.192.0 with Anchor IP 212.32.230.75, which belongs to server A.

Suppose we’ve developed an updated web application on server B and after months of testing, it’s finally ready.

To direct users visiting 89.149.192.0 to server B, we need to update the Anchor IP of Floating IP 89.149.192.0, changing (FLIP’ing) it from 212.32.230.75 (server A) to 212.32.230.66 (server B).

To do this manually, click  in the Customer Portal and change the Anchor IP:

Now, when you refresh your browser, the page from server B is shown:

Congratulations, you’ve just done a zero-downtime deployment, but also set your first step towards a high availability, continuous deployment web hosting cluster.

Step 2: Using the API to manage Floating IPs

Of course, using the Leaseweb Customer Portal is a convenient way to set up and play with Floating IPs, but the real power is in automation.

The official documentation of the Floating IPs API can be found on developer.leaseweb.com

In the following examples we’ll use curl to perform http requests and the jq tool to pretty-print the API responses, but you can use any tool or library for interacting with a RESTful API. You can find your API key (X-Lsw-Auth) in the Customer Portal under API

Floating IPs and Floating IP ranges have a prefix length and are always written in CIDR notation. In the context of API calls, the forward slash “/” is replaced with an underscore “_” for compatibility in URLs. For a single Floating IP address (/32), the prefix length may be omitted.

List Floating IP ranges

To list Floating IP ranges, make a GET request to /floatingIps/v2/ranges:
curl --silent --request GET --url https://api.leaseweb.com/floatingIps/v2/ranges --header 'X-Lsw-Auth: 213423-2134234-234234-23424' |jq

{
  "ranges": [
    {
      "id": "89.149.192.0_29",
      "range": "89.149.192.0/29",
      "customerId": "12345678",
      "salesOrgId": "2000",
      "pop": "AMS-01"
    }
  ],
  "_metadata": {
    "limit": 20,
    "offset": 0,
    "totalCount": 1
  }
}

List the Floating IP definitions in a Floating IP range

To list the Floating IP definitions within a certain Floating IP range, make a GET request to 

{
  "floatingIpDefinitions": [
    {
      "id": "89.149.192.0",
      "rangeId": "89.149.192.0_29",
      "pop": "AMS-01",
      "customerId": "12345678",
      "salesOrgId": "2000",
      "floatingIp": "89.149.192.0/32",
      "anchorIp": "212.32.230.66",
      "status": "ACTIVE",
      "createdAt": "2019-06-17T14:15:11+00:00",
      "updatedAt": "2019-06-26T09:26:52+00:00"
    }
  ],
  "_metadata": {
    "totalCount": 1,
    "limit": 20,
    "offset": 0
  }

{
  "id": "89.149.192.0",
  "rangeId": "89.149.192.0_29",
  "pop": "AMS-01",
  "customerId": "12345678",
  "salesOrgId": "2000",
  "floatingIp": "89.149.192.3/32",
  "anchorIp": "212.32.230.66",
  "status": "CREATING",
  "createdAt": "2019-06-26T14:30:40+00:00",
  "updatedAt": "2019-06-26T14:30:40+00:00"
}

{
  "floatingIpDefinitions": [
    {
      "id": "89.149.192.0",
      "rangeId": "89.149.192.0_29",
      "pop": "AMS-01",
      "customerId": "12345678",
      "salesOrgId": "2000",
      "floatingIp": "89.149.192.0/32",
      "anchorIp": "212.32.230.66",
      "status": "ACTIVE",
      "createdAt": "2019-06-17T14:15:11+00:00",
      "updatedAt": "2019-06-26T14:23:58+00:00"
    },
    {
      "id": "89.149.192.3",
      "rangeId": "89.149.192.0_29",
      "pop": "AMS-01",
      "customerId": "12345678",
      "salesOrgId": "2000",
      "floatingIp": "89.149.192.3/32",
      "anchorIp": "212.32.230.66",
      "status": "ACTIVE",
      "createdAt": "2019-06-26T14:30:40+00:00",
      "updatedAt": "2019-06-26T14:30:45+00:00"
    }
  ],
  "_metadata": {
    "totalCount": 2,
    "limit": 20,
    "offset": 0
  }
}

 updating 89.149.192.0 with Anchor IP 212.32.230.75, so we’re directing traffic back to server A again:
curl --silent --request PUT --url https://api.leaseweb.com/floatingIps/v2/ranges/89.149.192.0_29/floatingIpDefinitions/89.149.192.0_32--header 'X-Lsw-Auth: ' --header 'content-type: application/json' --data '{
    "anchorIp": "212.32.230.75"
}' |jq

{
  "id": "89.149.192.0",
  "rangeId": "89.149.192.0_29",
  "pop": "AMS-01",
  "customerId": "12345678",
  "salesOrgId": "2000",
  "floatingIp": "89.149.192.0/32",
  "anchorIp": "212.32.230.66",
  "status": "UPDATING",
  "createdAt": "2019-06-17T14:15:11+00:00",
  "updatedAt": "2019-06-26T14:35:57+00:00"
}

Note that in the response, the old anchorIP is still listed and the status has changed to UPDATING. The update process is very fast, but not instantaneous. When making another GET request to , you can see that the update has processed seconds later:

{
  "floatingIpDefinitions": [
    {
      "id": "89.149.192.0",
      "rangeId": "89.149.192.0_29",
      "pop": "AMS-01",
      "customerId": "12345678",
      "salesOrgId": "2000",
      "floatingIp": "89.149.192.0/32",
      "anchorIp": "212.32.230.75",
      "status": "ACTIVE",
      "createdAt": "2019-06-17T14:15:11+00:00",
      "updatedAt": "2019-06-26T14:36:01+00:00"
    }
  ],
  "_metadata": {
    "totalCount": 1,
    "limit": 20,
    "offset": 0
  }
}

Delete a Floating IP definition

Deleting a Floating IP definition is as easy as making a DELETE call to :
curl --silent --request DELETE --url https://api.leaseweb.com/floatingIps/v2/ranges/89.149.192.0_29/floatingIpDefinitions/89.149.192.3--header 'X-Lsw-Auth: ' |jq

{
  "id": "89.149.192.3",
  "rangeId": "89.149.192.0_29",
  "pop": "AMS-01",
  "customerId": "12345678",
  "salesOrgId": "2000",
  "floatingIp": "89.149.192.3/32",
  "anchorIp": "212.32.230.66",
  "status": "REMOVING",
  "createdAt": "2019-06-26T14:30:40+00:00",
  "updatedAt": "2019-06-26T14:39:34+00:00"
}

Just like with the POST and PUT calls, it will take a couple of seconds to process.

Step three: Putting it all together – creating a highly available web hosting platform with Keepalived

Keepalived is a versatile piece of software that can be used to implement automatic failover using the Leaseweb Floating IPs API. We’ll demonstrate how to create a simple active/backup setup where the Floating IP is automatically routed to server B in the event that server A fails.

It can do many more things, and keep in mind this is meant as a proof-of-concept example only, meant to demonstrate the how to be highly available with automatic failover and Floating IPs in the simplest possible way.

The keepalived configuration

After installing, the configuration of keepalived resides in the /etc/keepalived/keepalived.conf file. In this file, we’ll instruct keepalived to:

  • Create a “vrrp” instance named webservers with id 123:
    Note: the id can be any random number between 0-255, but it needs to be the same between all servers.
    vrrp_instance webservers { ... }
    virtual_router_id
  • Setup server A to be the master, with priority 200:
    state MASTER
    priority 200
  • Setup server B to be the backup, with priority 100:
    state BACKUP
    priority 100
  • Communicate with each other using a shared secret:
    interface <interface name> (see the instructions under Setting up the Floating IP address on the servers)
    unicast_src_IP <server's IP address>
    unicast_peer { <other server's IP address> }
    authentication { ... }
  • Run a script to update the Anchor IP when either server becomes master
    notify_master /etc/keepalived/becomemaster.sh
  • Run a command to check if the web server is still running. On server A (CentOS) this is the httpd process, on server B (Ubuntu), this is the nginx process and we need to wrap the command in a small script instead.
    track_script { ... }

So, we run the following commands to setup server A:

# Install keepalived
yum install -y keepalived

# Write keepalived config
cat <<EOF > /etc/keepalived/keepalived.conf
vrrp_instance webservers {
    virtual_router_id 123
    state MASTER
    priority 200
    interface eno1
    unicast_src_ip 212.32.230.75
    unicast_peer {
        212.32.230.66
    }
    authentication {
        auth_type PASS
        auth_pass supersecret
    }
    notify_master /etc/keepalived/becomemaster.sh
    track_script {
        chk_apache
    }
}

vrrp_script chk_apache {
    script "/usr/sbin/pidof httpd"
    interval 2
}
EOF

# Write script that calls floating IP API to update the Floating IP with this server as Anchor IP
cat <<EOF > /etc/keepalived/becomemaster.sh
#!/bin/sh
curl --silent --request PUT --url https://api.leaseweb.com/floatingIps/v2/ranges/89.149.192.0_29/floatingIpDefinitions/89.149.192.0_32 --header 'X-Lsw-Auth: '"213423-2134234-234234-23424" --header 'content-type: application/json' --data '{ "anchorIp": "212.32.230.75" }'
EOF
chmod +x /etc/keepalived/becomemaster.sh

# Restart keepalived
systemctl restart keepalived

# Check keepalived status
systemctl status keepalived

Result:

tim@laptop:~$ ssh root@20483.lsw
[root@servera ~]# yum install -y keepalived

[...]

[root@servera ~]# cat <<EOF > /etc/keepalived/keepalived.conf
> vrrp_instance webservers {
>     virtual_router_id 123
>     state MASTER
>     priority 200
>     interface eno1
>     unicast_src_ip 212.32.230.75
>     unicast_peer {
>         212.32.230.66
>     }
>     authentication {
>         auth_type PASS
>         auth_pass supersecret
>     }
>     notify_master /etc/keepalived/becomemaster.sh
>     track_script {
>         chk_apache
>     }
> }
>
> vrrp_script chk_apache {
>     script "/usr/sbin/pidof httpd"
>     interval 2
> }
> EOF

[root@servera ~]# cat <<EOF > /etc/keepalived/becomemaster.sh
> #!/bin/sh
> curl --silent --request PUT --url https://api.leaseweb.com/floatingIps/v2/ranges/89.149.192.0_29/floatingIpDefinitions/89.149.192.0_32 --header 'X-Lsw-Auth: '"213423-2134234-234234-23424" --header 'content-type: application/json' --data '{ "anchorIp": "212.32.230.75" }'
> EOF

[root@servera ~]# chmod +x /etc/keepalived/becomemaster.sh

[root@servera ~]# systemctl restart keepalived

[root@servera ~]# systemctl status keepalived
● keepalived.service - LVS and VRRP High Availability Monitor
   Loaded: loaded (/usr/lib/systemd/system/keepalived.service; disabled; vendor preset: disabled)
   Active: active (running) since Tue 2019-07-23 11:27:03 UTC; 30s ago
  Process: 1346 ExecStart=/usr/sbin/keepalived $KEEPALIVED_OPTIONS (code=exited, status=0/SUCCESS)
 Main PID: 1347 (keepalived)
   CGroup: /system.slice/keepalived.service
           ├─1347 /usr/sbin/keepalived -D
           ├─1348 /usr/sbin/keepalived -D
           └─1349 /usr/sbin/keepalived -D

Jul 23 11:27:03 servera Keepalived_vrrp[1349]: Opening file '/etc/keepalived/keepalived.conf'.
Jul 23 11:27:03 servera Keepalived_vrrp[1349]: WARNING - default user 'keepalived_script' for script execution does not exist ...reate.
Jul 23 11:27:03 servera Keepalived_vrrp[1349]: Truncating auth_pass to 8 characters
Jul 23 11:27:03 servera Keepalived_vrrp[1349]: SECURITY VIOLATION - scripts are being executed but script_security not enabled.
Jul 23 11:27:03 servera Keepalived_vrrp[1349]: Using LinkWatch kernel netlink reflector...
Jul 23 11:27:03 servera Keepalived_vrrp[1349]: VRRP sockpool: [ifindex(2), proto(112), unicast(1), fd(10,11)]
Jul 23 11:27:03 servera Keepalived_vrrp[1349]: VRRP_Script(chk_apache) succeeded
Jul 23 11:27:04 servera Keepalived_vrrp[1349]: VRRP_Instance(webservers) Transition to MASTER STATE
Jul 23 11:27:05 servera Keepalived_vrrp[1349]: VRRP_Instance(webservers) Entering MASTER STATE
Jul 23 11:27:05 servera Keepalived_vrrp[1349]: Opening script file /etc/keepalived/becomemaster.sh
Hint: Some lines were ellipsized, use -l to show in full.

[root@servera ~]#

Then we setup server B:

# Install keepalived
apt install -y keepalived

# Write keepalived config
cat <<EOF > /etc/keepalived/keepalived.conf
vrrp_script chk_nginx {
    script "/etc/keepalived/chk_nginx.sh"
    interval 2
}

vrrp_instance webservers {
    virtual_router_id 123
    state BACKUP
    priority 100
    interface enp32s0
    unicast_src_ip 212.32.230.66
    unicast_peer {
        212.32.230.75
    }
    authentication {
        auth_type PASS
        auth_pass supersecret
    }
    notify_master /etc/keepalived/becomemaster.sh
    track_script {
        chk_nginx
    }
}
EOF

# Write script that calls floating IP API to update the Floating IP with this server as Anchor IP
cat <<EOF > /etc/keepalived/becomemaster.sh
#!/bin/sh
curl --silent --request PUT --url https://api.leaseweb.com/floatingIps/v2/ranges/89.149.192.0_29/floatingIpDefinitions/89.149.192.0_32 --header 'X-Lsw-Auth: '"213423-2134234-234234-23424" --header 'content-type: application/json' --data '{ "anchorIp": "212.32.230.66" }'
EOF
chmod +x /etc/keepalived/becomemaster.sh

# Restart keepalived
systemctl restart keepalived

# Check keepalived status
systemctl status keepalived

Result:

tim@laptop:~$ ssh root@37089.lsw
[root@serverb ~]# apt install -y keepalived

[...]

[root@serverb ~]# cat <<EOF > /etc/keepalived/keepalived.conf
> vrrp_instance webservers {
>     virtual_router_id 123
>     state BACKUP
>     priority 100
>     interface enp32s0
>     unicast_src_ip 212.32.230.66
>     unicast_peer {
>         212.32.230.75
>     }
>
>     authentication {
>         auth_type PASS
>         auth_pass supersecret
>     }
>
>     notify_master /etc/keepalived/becomemaster.sh
>
>     track_script {
>         chk_nginx
>     }
> }
>
> vrrp_script chk_nginx {
>     script "/etc/keepalived/chk_nginx.sh"
>     interval 2
> }
> EOF

[root@serverb ~]# cat <<EOF > /etc/keepalived/becomemaster.sh
> #!/bin/sh
> curl --silent --request PUT --url https://api.leaseweb.com/floatingIps/v2/ranges/89.149.192.0_29/floatingIpDefinitions/89.149.192.0_32 --header 'X-Lsw-Auth: '"213423-2134234-234234-23424" --header 'content-type: > application/json' --data '{ "anchorIp": "212.32.230.66" }'
> EOF

[root@serverb ~]# cat <<EOF > /etc/keepalived/chk_nginx.sh
> #!/bin/sh
> /bin/pidof nginx
> EOF

[root@serverb ~]# chmod +x /etc/keepalived/becomemaster.sh

[root@serverb ~]# systemctl restart keepalived

[root@serverb ~]# systemctl status keepalived
● keepalived.service - Keepalive Daemon (LVS and VRRP)
   Loaded: loaded (/lib/systemd/system/keepalived.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2019-07-23 11:27:12 UTC; 48s ago
  Process: 24346 ExecStart=/usr/sbin/keepalived $DAEMON_ARGS (code=exited, status=0/SUCCESS)
 Main PID: 24355 (keepalived)
    Tasks: 3 (limit: 4574)
   CGroup: /system.slice/keepalived.service
           ├─24355 /usr/sbin/keepalived
           ├─24357 /usr/sbin/keepalived
           └─24358 /usr/sbin/keepalived

Jul 23 11:27:12 serverb Keepalived_vrrp[24358]: Registering Kernel netlink command channel
Jul 23 11:27:12 serverb Keepalived_vrrp[24358]: Registering gratuitous ARP shared channel
Jul 23 11:27:12 serverb Keepalived_vrrp[24358]: Opening file '/etc/keepalived/keepalived.conf'.
Jul 23 11:27:12 serverb Keepalived_vrrp[24358]: WARNING - default user 'keepalived_script' for script execution does not exist - please create.
Jul 23 11:27:12 serverb Keepalived_vrrp[24358]: Truncating auth_pass to 8 characters
Jul 23 11:27:12 serverb Keepalived_vrrp[24358]: SECURITY VIOLATION - scripts are being executed but script_security not enabled.
Jul 23 11:27:12 serverb Keepalived_vrrp[24358]: Using LinkWatch kernel netlink reflector...
Jul 23 11:27:12 serverb Keepalived_vrrp[24358]: VRRP_Instance(webservers) Entering BACKUP STATE
Jul 23 11:27:12 serverb Keepalived_healthcheckers[24357]: Opening file '/etc/keepalived/keepalived.conf'.
Jul 23 11:27:12 serverb Keepalived_vrrp[24358]: VRRP_Script(chk_nginx) succeeded

[root@serverb ~]# 

Watching keepalived in action

So now that we have our redundant setup and server A is the master. If we visit the Floating IP address in our browser, we see that it’s being served from server A:

Let’s simulate a failure on server A by shutting down the Apache web server with the and watch server B take over.

On server A, run:
systemctl stop httpd

Within a couple of seconds, you’ll see it failover to server B. Feel free to hammer F5 like your life depends on it!

Looking at the logs of keepalived on server B, you can see that it detected the failure on server A and automatically executed the script to update the Anchor IP:

journalctl -u keepalived |tail

[ ... ]

Jul 23 11:51:43 diy-dhcp-ams01-nl Keepalived_vrrp[24358]: VRRP_Instance(webservers) Transition to MASTER STATE
Jul 23 11:51:44 diy-dhcp-ams01-nl Keepalived_vrrp[24358]: VRRP_Instance(webservers) Entering MASTER STATE
Jul 23 11:51:44 diy-dhcp-ams01-nl Keepalived_vrrp[24358]: Opening script file /etc/keepalived/becomemaster.sh

That’s it, you now have your own (minimal implementation of) a highly available web hosting platform!

Share

Distributed storage for HA clusters (based on GlusterFS)

Nowadays everyone is searching for HA (High Available) solutions for running more powerful applications or websites. As you can see we already have post on our blog about configuring loadbalancers on Ubuntu (http://www.leaseweb.com/labs/2011/09/setting-up-keepalived-on-ubuntu-load-balancing-using-haproxy-on-ubuntu-part-2/) i will expand that post with the possibility to run highly available and balanced system which you can use for your shared hosting or high traffic website without having a single point of failure.

The actual problem of balanced / clustered solutions often the content server where you keep all data, that can be: databases, static files, uploaded files. All this content needs to be distributed across all your servers and you’ll have to keep track on modifications, which is not really convenient.  In other case you will have a single point of failure of your balanced solution. You can use rsync or lsyncd for doing this, but to make things more simple from administrative perspective and get more advantage for future, you can use a DFS (distributed file system), nowadays there are plenty suitable open source options like: ocfs, glusterfs, mogilefs, pvfs2, chironfs, xtreemfs etc.

Why we want to use DFS?

  • Data sharing of multiple users
  • user mobility
  • location transparency
  • location independence
  • backups and centralized management

For this post, I choose GlusterFS to start with. Why? It is opensource, it has modular, plug-able interface and you can run it on any linux based server without upgrading your kernel. Maybe i will make some overview of others DFS in one of my next blog posts.

We will use 3 dedicated servers like HP120G6:

1. HP120G6 / 1 x QC X3440 CPU / 2 x 1Gbit NICs / 4GB RAM / 2 x 1TB SATA2 (disks you can add later on fly :))

Let’s assume that we already have dedicated server with CentOS 6 installed on the first 1TB hard drive.

Just ssh to your host prepare HDDs and install GlusterFS server with agent.

#ssh root@85.17.xxx.xx

Now we need to prepare HDDs, we will use second 1TB drive for distributed storage:

#parted /dev/sdb
mklabel gpt
unit TB
mkpart primary 0 1TB
print
quit
mkfs.ext4 /dev/sdb1

After that we will install all required packages for gluster:

#yum -y install wget fuse fuse-libs automake bison gcc flex libtool
#yum -y install compat-readline5 compat-libtermcap
#wget http://packages.sw.be/rsync/rsync-3.0.7-1.el5.rfx.x86_64.rpm<

Let’s download the gluster packages from their website:

#wget http://download.gluster.com/pub/gluster/glusterfs/LATEST/CentOS/glusterfs-core-3.2.4-1.x86_64.rpm
#wget http://download.gluster.com/pub/gluster/glusterfs/LATEST/CentOS/glusterfs-fuse-3.2.4-1.x86_64.rpm
#wget http://download.gluster.com/pub/gluster/glusterfs/LATEST/CentOS/glusterfs-geo-replication-3.2.4-1.x86_64.rpm
#wget http://download.gluster.com/pub/gluster/glusterfs/LATEST/CentOS/glusterfs-rdma-3.2.4-1.x86_64.rpm
#rpm -Uvh glusterfs-core-3.2.4-1.x86_64.rpm
#rpm -Uvh glusterfs-geo-replication-3.2.4-1.x86_64.rpm

We also want to run our storage on a separated network card, let’s configure eth1 for that using internal IP range:

#nano /etc/sysconfig/network-scripts/ifcfg-eth1
DEVICE="eth1"
HWADDR="XX:XX:XX:XX:XX:XX"
NM_CONTROLLED="yes"
BOOTPROTO="static"
IPADDR="10.0.10.X"
NETMASK="255.255.0.0"
ONBOOT="yes"

We also want to use hostnames while configuring gluster:

#nano /etc/hosts
10.0.10.1       gluster1-server
10.0.10.2       gluster2-server
10.0.10.3       gluster3-server

Ensure that TCP ports 111, 24007,24008, 24009-(24009 + number of bricks across all volumes) are open on all Gluster servers.

You can use the following chains with iptables:

#iptables -A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 24007:24047 -j ACCEPT
#iptables -A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 111 -j ACCEPT
#iptables -A RH-Firewall-1-INPUT -m state --state NEW -m udp -p udp --dport 111 -j ACCEPT
#iptables -A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 38465:38467 -j ACCEPT
#service iptables save
#service iptables restart

Now check the version of installed glusterfs:

#/usr/sbin/glusterfs -V

To configure Red Hat-based systems to automatically start the glusterd daemon every time the system boots, enter the following from the command line:

#chkconfig glusterd on

GlusterFS offers a single command line utility known as the Gluster Console Manager to simplify configuration and management of your storage environment. The Gluster Console Manager provides functionality similar to the LVM (Logical Volume Manager) CLI or ZFS Command Line Interface, but across multiple storage servers. You can use the Gluster Console Manager online, while volumes are mounted and active.

You can run the Gluster Console Manager on any Gluster storage server. You can run Gluster commands either by invoking the commands directly from the shell, or by running the Gluster CLI in interactive mode.

To run commands directly from the shell, for example:

#gluster peer status

To run the Gluster Console Manager in interactive mode:

#gluster

Upon invoking the Console Manager, you will get an interactive shell where you can execute gluster commands, for example:

gluster > peer status

Before configuring a GlusterFS volume, you need to create a trusted storage pool consisting of the storage servers that will make up the volume. A storage pool is a trusted network of storage servers. When you start the first server, the storage pool consists of that server alone. To add additional storage servers to the storage pool, you can use the probe command from a storage server.

To add servers to the trusted storage pool use following command for each server you have, example:

gluster peer probe gluster2-server
gluster peer probe gluster3-server

Verify the peer status from the first server using the following commands:

# gluster peer probe gluster1-server
Number of Peers: 2

Hostname: gluster2-server
Uuid: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
State: Peer in Cluster (Connected)

Hostname: gluster3-server
Uuid: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
State: Peer in Cluster (Connected)

This way, you can add additional storage servers to your storage pool on the fly, to remove a server to the storage pool, use the following command:

# gluster peer detach gluster3-server
Detach successful

Now we can create a replicated volume.
A volume is a logical collection of bricks where each brick is an export directory on a server in the trusted storage pool. Most of the gluster management operations happen on the volume.
Replicated volumes replicate files throughout the bricks in the volume. You can use replicated volumes in environments where high-availability and high-reliability are critical.

First we ssh to each server we have to create folders and mount HDDs:

#mkdir /storage1
mount /dev/sdb1 /storage1

#mkdir /storage2
mount /dev/sdb1 /storage2

#mkdir /storage3
mount /dev/sdb1 /storage3

Create the replicated volume using the following command:

# gluster volume create test-volume replica 3 transport tcp gluster1-server:/storage1 gluster2-server:/storage2 gluster3-server:/storage3
Creation of test-volume has been successful
Please start the volume to access data.

You can optionally display the volume information using the following command:

# gluster volume info
Volume Name: test-volume
Type: Distribute
Status: Created
Number of Bricks: 3
Transport-type: tcp
Bricks:
Brick1: gluster1-server:/storage1
Brick2: gluster2-server:/storage2
Brick3: gluster3-server:/storage3

You must start your volumes before you try to mount them, start the volume using the following command:

# gluster volume start test-volume
Starting test-volume has been successful

Now we need to setup GlusterFS client to access our volume, Gluster offers multiple options to access gluster volumes:

  • Gluster Native Client – This method provides high concurrency, performance and transparent failover in GNU/Linux clients. The Gluster Native Client is POSIX conformant. For accessing volumes using gluster native protocol, you need to install gluster native client.
  • NFS – This method provides access to gluster volumes with NFS v3 or v4.
  • CIFS – This method provides access to volumes when using Microsoft Windows as well as SAMBA clients. For this access method, Samba packages need to be present on the client side.

We will talk about Gluster Native Client as it is a POSIX conformant, FUSE-based, client running in user space. Gluster Native Client is recommended for accessing volumes when high concurrency and high write performance is required.

Verify that the FUSE module is installed:

# modprobe fuse
# dmesg | grep -i fuse
fuse init (API version 7.XX)

Install required prerequisites on the client using the following command:

# sudo yum -y install openssh-server wget fuse fuse-libs openib libibverbs

Ensure that TCP and UDP ports 24007 and 24008 are open on all Gluster servers. Apart from these ports, you need to open one port for each brick starting from port 24009. For example: if you have five bricks, you need to have ports 24009 to 24014 open.

You can use the following chains with iptables:

# sudo iptables -A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 24007:24008 -j ACCEPT
# sudo iptables -A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 24009:24014 -j ACCEPT

Install Gluster Native Client (FUSE component) using following command:

#rpm -Uvh glusterfs-fuse-3.2.4-1.x86_64.rpm
#rpm -Uvh glusterfs-rdma-3.2.4-1.x86_64.rpm

After that we need to mount volumes we created before:

To manually mount a Gluster volume, use the following command on each server we have:

# mount -t glusterfs gluster1-server:/test-volume /mnt/glusterfs

You can configure your system to automatically mount the Gluster volume each time your system starts.
Edit the /etc/fstab file and add the following line on each server we have:

gluster1-server:/test-volume /mnt/glusterfs glusterfs defaults,_netdev 0 0

To test mounted volumes we can easily use command:

# df -h /mnt/glusterfs Filesystem Size Used Avail Use% Mounted on gluster1-server:/test-volume
1T 1.3G 1TB 1% /mnt/glusterfs

Now you can use /mnt/glusterfs mount as HA storage for your files, backups or even XEN virtualization and you don’t need to worry if 1 server is going down because of hardware problem or you planned some maintenance on it.

Next time, i will write about how to integrate Gluster with NFS without having a single point of failure, so you can use it with any solution which supports the NFS protocol.

Any suggestions and comments are appreciated,Thank you!

Share

Setting up keepalived on Ubuntu (load balancing using HAProxy on Ubuntu part 2)

In our previous post we have set up a HAProxy loadbalancer to balance the load of our web application between three webservers, here’s the diagram of the situation we have ended up with:

              +---------+
              |  uplink |
              +---------+
                   |
                   +
                   |
              +---------+
              | loadb01 |
              +---------+
                   |
     +-------------+-------------+
     |             |             |
+---------+   +---------+   +---------+
|  web01  |   |  web02  |   |  web03  |
+---------+   +---------+   +---------+

As we already concluded in the last post, there’s still a single point of failure in this setup. If the loadbalancer dies for some reason the whole site will be offline. In this post we will add a second loadbalancer and setup a virtual IP address shared between the loadbalancers. The setup will look like this:

              +---------+
              |  uplink |
              +---------+
                   |
                   +
                   |
+---------+   +---------+   +---------+
| loadb01 |---|virtualIP|---| loadb02 |
+---------+   +---------+   +---------+
                   |
     +-------------+-------------+
     |             |             |
+---------+   +---------+   +---------+
|  web01  |   |  web02  |   |  web03  |
+---------+   +---------+   +---------+

So our setup now is:
– Three webservers, web01 (192.168.0.1), web02 (192.168.0.2 ), and web03 (192.168.0.3) each serving the application
– The first load balancer (loadb01, ip: (192.168.0.100 ))
– The second load balancer (loadb02, ip: (192.168.0.101 )), configure this in the same way as we configured the first one.

To setup the virtual IP address we will use keepalived (als also suggested by Warren in the comments):

loadb01$ sudo apt-get install keepalived

Good, keepalived is now installed. Before we proceed with configuring keepalived itself, edit the following file:

loadb01$ sudo vi /etc/sysctl.conf

And add this line to the end of the file:

net.ipv4.ip_nonlocal_bind=1

This option is needed for applications (haproxy in this case) to be able to bind to non-local addresses (ip adresses which do not belong to an interface on the machine). To apply the setting, run the following command:

loadb01$ sudo sysctl -p

Now let’s add the configuration for keepalived, open the file:

loadb01$ sudo vi /etc/keepalived/keepalived.conf

And add the following contents (see comments for details ont he configuration!):

# Settings for notifications
global_defs {
    notification_email {
        your@emailaddress.com			# Email address for notifications 
    }
    notification_email_from loadb01@domain.ext  # The from address for the notifications
    smtp_server 127.0.0.1			# You can specifiy your own smtp server here
    smtp_connect_timeout 15
}
 
# Define the script used to check if haproxy is still working
vrrp_script chk_haproxy { 
    script "killall -0 haproxy" 
    interval 2 
    weight 2 
}
 
# Configuation for the virtual interface
vrrp_instance VI_1 {
    interface eth0
    state MASTER 				# set this to BACKUP on the other machine
    priority 101				# set this to 100 on the other machine
    virtual_router_id 51
 
    smtp_alert					# Activate email notifications
 
    authentication {
        auth_type AH
        auth_pass myPassw0rd			# Set this to some secret phrase
    }
 
    # The virtual ip address shared between the two loadbalancers
    virtual_ipaddress {
        192.168.0.200
    }
    
    # Use the script above to check if we should fail over
    track_script {
        chk_haproxy
    }
}

And start keepalived:

loadb01$ /etc/init.d/keepalived start

Now the next step is to install and configure keepalived on our second loadbalancer aswell, redo the steps starting from apt-get install keepalived. In the configuration step for keepalived, be sure change these two settings:

    state MASTER 				# set this to BACKUP on the other machine
    priority 101				# set this to 100 on the other machine

To:

    state BACKUP 			
    priority 100			

That’s it! We have now configured a virtual IP shared between our two loadbalancers, you can try loading the haproxy statistic page on the virtual IP adddress and should get the statistics for loadb01, then switch off loadb01 and refresh, the virtual IP address will now be assigned to the second loadbalancer and you should see the statistics page for that.

In a next post we will focus on adding MySQL to this setup as requested by Miquel in the comments on the previous post in this series. If there’s anything else you’d like us to cover, or if you have any questions please leave a comment!

Share