I am a newbie to CentOS/Linux/HAProxy/Keepalived, so I have had a lot to learn over the past few days!

I have 2 CentOS 6.5 servers in Amazon AWS running on their own EC2 Micro instances. I also have 1 Elastic IP set up. Each instance has the following installed:

  • Amazon EC2 API Tools
  • HAProxy
  • Keepalived
  • Apache / httpd
  • Java OpenJDK 1.6

I have got HAProxy installed and configured on both instances. For testing purposes I installed Apache to serve a basic index.html page so I could test that HAProxy was carrying out the LB task and round robin is working fine.

What I am trying to achieve, is using Keepalived to monitor the haproxy process on LB1 (Has EIP assigned), if it fails or stops for some reason, it calls a script which uses the AWS EC2 API Tools to reassign an Elastic IP to the second Loadbalancer to take over..

I have googled and looked at the documentation to create a script that achieves this, however, when I test it and the HAProxy service is stopped, it does not run the script..

LB1 - Contents of keepalived.conf

vrrp_script chk_haproxy {
script "killall -0 haproxy"
interval 2
weight 2

vrrp_instance VI_1 {
interface eth0
state MASTER
virtual_router_id 51
priority 101
vrrp_unicast_bind 10.xx.xx.01 # The IP of the host this file is stored on
vrrp_unicast_peer 10.xx.xx.23 # The IP of the other host
advert_int 1
track_script {
notify_fault /etc/keepalived/vrrp.sh

Output from Tailing the log

Jun 18 12:03:11 ip-10-xx-xx-xx Keepalived_vrrp[25199]: Opening file '/etc/keepalived/keepalived.conf'.
Jun 18 12:03:11 ip-10-xx-xx-xx Keepalived_vrrp[25199]: Configuration is using : 60618 Bytes
Jun 18 12:03:11 ip-10-xx-xx-xx Keepalived_vrrp[25199]: Using LinkWatch kernel netlink reflector...
Jun 18 12:03:11 ip-10-xx-xx-xx Keepalived_vrrp[25199]: VRRP sockpool: [ifindex(2), proto(112), fd(10,11)]
Jun 18 12:03:11 ip-10-xx-xx-xx Keepalived_vrrp[25199]: VRRP_Script(chk_haproxy) succeeded
Jun 18 12:03:12 ip-10-xx-xx-xx Keepalived_vrrp[25199]: VRRP_Instance(VI_1) Transition to MASTER STATE
Jun 18 12:03:13 ip-10-xx-xx-xx Keepalived_vrrp[25199]: VRRP_Instance(VI_1) Entering MASTER STATE
Jun 18 12:03:19 ip-10-xx-xx-xx Keepalived_vrrp[25199]: VRRP_Script(chk_haproxy) failed

My first impression is that it has detected the haproxy service failure, but it doesnt seem to execute the vrrp.sh script. (contents below)

cd $EC2_BASE/tools/bin
#DisAssociate EIP from this instance.
./ec2-disassociate-address  54.xx.xx.xx
#Mapping EIP to secondary server
./ec2-associate-address 54.xx.xx.xx -i [instance-id]

If I run the vrrp.sh script by itself on the server, it runs the commands and successfully re-assigns the EIP. Just not when it is called from the keepalived.conf script.

Any ideas!? I've been scratching my head all day and the laptop is about to be launched out the window..

  • In scripts the first line must be the interpreter switch the lines #vrrp.sh and #!/bin/bash so the shebang becomes the first line.
    – HBruijn
    Nov 25 '15 at 9:51

The notify_fault script is invoked when keepalived enters the fault state.

You want to configure your script using

notify_master /etc/keepalived/vrrp.sh
  • I tried this also, but it still failed to run the script. So if Keepalived enters the master state it would then run the script which would reassign the script to the other load balancer? Slave? I think i'm confusing myself now.. :-/
    – rosey85uk
    Jun 19 '14 at 12:51
  • Can you verify this from the logs? A desparate try could be to make it the general notify script, and check the first command line argument to determine if the current transition is to MASTER state. Jun 19 '14 at 12:54
  • My Logs: [root@ip-xx.xx.xx.xx ~]# tail /var/log/messages Jun 19 13:13:53 ip-xx.xx.xx.xx Keepalived_vrrp[19601]: VRRP sockpool: [ifindex(2), proto(112), fd(10,11)] Jun 19 13:13:54 ip-xx.xx.xx.xx Keepalived_vrrp[19601]: VRRP_Instance(VI_1) Transition to MASTER STATE Jun 19 13:13:55 ip-xx.xx.xx.xx Keepalived_vrrp[19601]: VRRP_Instance(VI_1) Entering MASTER STATE Jun 19 13:13:59 ip-xx.xx.xx.xx Keepalived_vrrp[19601]: VRRP_Script(chk_haproxy) succeeded Jun 19 14:10:30 ip-xx.xx.xx.xx Keepalived_vrrp[19601]: VRRP_Script(chk_haproxy) failed
    – rosey85uk
    Jun 19 '14 at 15:49

Can you please paste your backup.conf and slave.conf files?

Also, you should check these two articles:

http://www.trk7.com/blog/keepalived-instance-not-entering-failed-state/ or http://comments.gmane.org/gmane.linux.keepalived.devel/4102 ?


First to determine if the script is or is not executing, I would add the following line directly below #!/bin/bash.

echo "----------vrrp.sh running ------" >> /var/log/messages 

or something whatever you want... Just to log the fact that the application has seen the "state" transition from backup to master or whatever, and has in fact executed the script as intended.

You could put it at the end of the script as well, the point is to just make sure the the script was called.

Test it again, and then cat /var/log/messages, and look for your echo... you should see it, if you don't then it could be any number of things, permissions, user account that is running, ensure the script executable, and so on... likely based on your initial comment I suspect it is executing its just not doing what you expect.

So if you do see your echo, as I expect you will, then I would look at your environment configuration, what you could do, just for testing, is export your JAVA_HOME along with your EC2 variables in the script... add them after the echo line you just added. Obviously this is just for testing purposes and once you confirm its all working as expected, then you should remove this from the script, and configure your environment appropriately. My final suggestion here is to change your command... You can disassociate and associate the EIP in a single command.

So now your script will look something like this:


echo "----------vrrp.sh running ------" >> /var/log/messages

cd /usr/local/ec2/ec2-api-tools-

export JAVA_HOME="/usr/lib/jvm/jre-1.8.0-openjdk-"

export EC2_BASE="/usr/local/ec2/ec2-api-tools-"
export EC2_URL=https://ec2.us-east-1.amazonaws.com
export AWS_SECRET_KEY=InBiqtOCfaJhCb3u3jE6gpmkW5shgjP8N++/7Huv

#Those are obviously not my access/secret keys, just providing an example

./ec2-associate-address -a eipalloc-s234523 -i i-023ksdfj --allow-reassociation

Final note:

Be sure that the IAM User account you are authenticating with has the necessary IAM user account policy attached... an example of a policy is below you will have to go to policies to create it, and then once created just got to the users and the specific IAM user account you are authenticating with and attach the policy you just created unless you are using an FULL ACCESS account in which case, I would recommend against that. and to setup an IAM user account with the policy outlined below:

arn:aws:iam::107890181863:policy/MOVE-EIP or whatever.

"Version": "2015-1-1",
"Statement": [
        "Action": [
        "Effect": "Allow",
        "Resource": "*"

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.