NAME
carp – Common Address Redundancy Protocol
SYNOPSIS
pseudo-device carp
DESCRIPTION
The carp interface is a pseudo-device which implements and controls the
CARP protocol. carp allows multiple hosts on the same local network to
share a set of IP addresses. Its primary purpose is to ensure that these
addresses are always available, but in some configurations carp can also
provide load balancing functionality.
A carp interface can be created at runtime using the ifconfig carpN
create command or by setting up a hostname.if(5) configuration file for
netstart(8).
To use carp, the administrator needs to configure at minimum a common
virtual host ID (VHID) and virtual host IP address on each machine which
is to take part in the virtual group. Additional parameters can also be
set on a per-interface basis: advbase and advskew, which are used to con-
trol how frequently the host sends advertisements when it is the master
for a virtual host, and pass which is used to authenticate carp adver-
tisements. Finally carpdev is used to specify which interface the carp
device attaches to. If unspecified, the kernel attempts to set it by
looking for another interface with the same subnet. These configurations
can be done using ifconfig(8), or through the SIOCSVH ioctl.
carp can also be used in conjunction with ifstated(8) to respond to
changes in CARP state; however, for most uses this will not be necessary.
See the manual page for ifstated(8) for more information.
Additionally, there are a number of global parameters which can be set
using sysctl(8):
net.inet.carp.allow Accept incoming carp packets. Enabled by de-
fault.
net.inet.carp.preempt Allow virtual hosts to preempt each other.
It is also used to failover carp interfaces
as a group. When the option is enabled and
one of the carp enabled physical interfaces
goes down, advskew is changed to 240 on all
carp interfaces. See also the first example.
Disabled by default.
net.inet.carp.log Make carp log state changes, bad packets, and
other errors. May be a value between 0 and 7
corresponding with syslog(3) priorities. The
default value is 2, which limits logging to
changes in CARP state.
LOAD BALANCING
carp provides two mechanisms to load balance incoming traffic over a
group of carp hosts: ARP balancing and IP balancing.
Which one to use mainly depends on the network environment carp is being
used in. ARP balancing has limited abilities for load balancing the in-
coming connections between hosts in an Ethernet network. It only works
for clients in the local network, because ARP balancing spreads the load
by varying ARP replies based on the source MAC address of the host send-
ing the query. Therefore it cannot balance traffic that crosses a
router, because the router itself will always be balanced to the same
virtual host.
IP balancing is not dependent on ARP and therefore also works for traffic
that comes over a router. This method should work in all environments
and can also provide more fine grained load balancing than ARP balancing.
The downside of IP balancing is that it requires the traffic that is des-
tined towards the load balanced IP addresses to be received by all carp
hosts. While this is always the case when connected to a hub, it has to
play some tricks in switched networks, which will result in a higher net-
work load.
A rule of thumb might be to use ARP balancing if there are many hosts on
the same network segment and to use IP balancing for all other cases.
To configure load balancing one has to specify multiple carp nodes using
the carpnodes option. Each node in a load balancing cluster is repre-
sented by at least one “vhid:advskew” pair in a comma separated list.
carp tries to distribute the incoming network load over all configured
carpnodes. The following example creates a load balancing group consist-
ing of three nodes, using vhids 3, 4 and 6:
# ifconfig carp0 carpnodes 3:0,4:0,6:100
The advskew value of the last node is set to 100, so that this node is
designated to the BACKUP state. It will only become MASTER if all nodes
with a lower advskew value have failed. By varying this value throughout
the machines in the cluster it is possible to decide which share of the
network load each node receives. Therefore, all carp interfaces in the
cluster are configured identically, except for a different advskew value
within the carpnodes specification.
See the EXAMPLES section for a practical example of load balancing.
ARP BALANCING
For ARP balancing, one has to configure multiple carpnodes and choose the
balancing mode arp.
Once an ARP request is received, the CARP protocol will use a hashing
function against the source MAC address in the ARP request to determine
which carpnode the request belongs to. If the corresponding carpnode is
in master state, the ARP request will be answered, otherwise it will be
ignored.
The ARP load balancing has some limitations. Firstly, ARP balancing only
works on the local network segment. It cannot balance traffic that
crosses a router, because the router itself will always be balanced to
the same carpnode. Secondly, ARP load balancing can lead to asymmetric
routing of incoming and outgoing traffic, thus combining it with
pfsync(4) requires special care, because this can create a race condition
between balanced routers and the host they are serving. ARP balancing
can be safely used with pfsync if the pf(4) ruleset translates the source
address to an unshared address on the outgoing interface using a NAT
rule. This requires multiple CARP groups with different IP addresses on
the outgoing interface, configured so that each host is the master of one
group.
ARP balancing also works for IPv6, but instead of ARP the Neighbor Dis-
covery Protocol (NDP) is used.
IP BALANCING
IP load balancing works by utilizing the network itself to distribute in-
coming traffic to all carp nodes in the cluster. Each packet is filtered
on the incoming carp interface so that only one node in the cluster ac-
cepts the packet. All the other nodes will just silently drop it. The
filtering function uses a hash over the source and destination address of
the IPv4 or IPv6 packet and compares the result against the state of the
carpnode.
IP balancing is activated by setting the balancing mode to ip. This is
the recommended default setting. In this mode, carp uses a multicast MAC
address, so that a switch sends incoming traffic towards all nodes.
However, there are a few OS and routers that do not accept a multicast
MAC address being mapped to a unicast IP. This can be resolved by using
one of the following unicast options. For scenarios where a hub is used
it is not necessary to use a multicast MAC and it is safe to use the ip-
unicast mode. Manageable switches can usually be tricked into forwarding
unicast traffic to all cluster nodes ports by configuring them into some
sort of monitoring mode. If this is not possible, using the ip-stealth
mode is another option, which should work on most switches. In this mode
carp never sends packets with its virtual MAC address as source. Stealth
mode prevents a switch from learning the virtual MAC address, so that it
has to flood the traffic to all its ports. Please note that activating
stealth mode on a carp interface that has already been running might not
work instantly. As a workaround the VHID of the first carpnode can be
changed to a previously unused one, or just wait until the MAC table en-
try in the switch times out. Some Layer-3 switches do port learning
based on ARP packets. Therefore the stealth mode cannot hide the virtual
MAC address from these kind of devices.
If IP balancing is being used on a firewall, it is recommended to config-
ure the carpnodes in a symmetrical manner. This is achieved by simply
using the same carpnodes list on all sides of the firewall. This ensures
that packets of one connection will pass in and out on the same host and
are not routed asymmetrically.
EXAMPLES
For firewalls and routers with multiple interfaces, it is desirable to
failover all of the carp interfaces together, when one of the physical
interfaces goes down. This is achieved by the preempt option. Enable it
on both host A and B:
# sysctl net.inet.carp.preempt=1
Assume that host A is the preferred master and 192.168.1.x/24 is config-
ured on one physical interface and 192.168.2.y/24 on another. This is
the setup for host A:
# ifconfig carp0 192.168.1.1 vhid 1
# ifconfig carp1 192.168.2.1 vhid 2
The setup for host B is identical, but it has a higher advskew:
# ifconfig carp0 192.168.1.1 vhid 1 advskew 100
# ifconfig carp1 192.168.2.1 vhid 2 advskew 100
Because of the preempt option, when one of the physical interfaces of
host A fails, advskew is adjusted to 240 on all its carp interfaces.
This will cause host B to preempt on both interfaces instead of just the
failed one.
LOAD BALANCING
In order to set up a load balanced virtual host, it is necessary to con-
figure one carpnodes entry for each physical host. In the following ex-
ample, two physical hosts are configured to provide balancing and
failover for the IP address 192.168.1.10.
First the carp interface on Host A is configured. The advskew of 100 on
the second carpnode entry means that its advertisements will be sent out
slightly less frequently and will therefore become the designated backup.
# ifconfig carp0 192.168.1.10 carpnodes 1:0,2:100 balancing ip
The configuration for host B is identical, except the skew is on the
carpnode entry with virtual host 1 rather than virtual host 2.
# ifconfig carp0 192.168.1.10 carpnodes 1:100,2:0 balancing ip
If ARP balancing or a different mode of IP balancing is desired the
balancing mode can be adjusted accordingly.
SEE ALSO
sysctl(3), inet(4), pfsync(4), hostname.if(5), ifconfig(8), ifstated(8),
netstart(8), sysctl(8)
NAME carp – Common Address Redundancy Protocol
SYNOPSIS pseudo-device carp
DESCRIPTION The carp interface is a pseudo-device which implements and controls the CARP protocol. carp allows multiple hosts on the same local network to share a set of IP addresses. Its primary purpose is to ensure that these addresses are always available, but in some configurations carp can also provide load balancing functionality.
A carp interface can be created at runtime using the ifconfig carpN create command or by setting up a hostname.if(5) configuration file for netstart(8).
To use carp, the administrator needs to configure at minimum a common virtual host ID (VHID) and virtual host IP address on each machine which is to take part in the virtual group. Additional parameters can also be set on a per-interface basis: advbase and advskew, which are used to con- trol how frequently the host sends advertisements when it is the master for a virtual host, and pass which is used to authenticate carp adver- tisements. Finally carpdev is used to specify which interface the carp device attaches to. If unspecified, the kernel attempts to set it by looking for another interface with the same subnet. These configurations can be done using ifconfig(8), or through the SIOCSVH ioctl.
carp can also be used in conjunction with ifstated(8) to respond to changes in CARP state; however, for most uses this will not be necessary. See the manual page for ifstated(8) for more information.
Additionally, there are a number of global parameters which can be set using sysctl(8):
net.inet.carp.allow Accept incoming carp packets. Enabled by de- fault.
net.inet.carp.preempt Allow virtual hosts to preempt each other. It is also used to failover carp interfaces as a group. When the option is enabled and one of the carp enabled physical interfaces goes down, advskew is changed to 240 on all carp interfaces. See also the first example. Disabled by default.
net.inet.carp.log Make carp log state changes, bad packets, and other errors. May be a value between 0 and 7 corresponding with syslog(3) priorities. The default value is 2, which limits logging to changes in CARP state.
LOAD BALANCING carp provides two mechanisms to load balance incoming traffic over a group of carp hosts: ARP balancing and IP balancing.
Which one to use mainly depends on the network environment carp is being used in. ARP balancing has limited abilities for load balancing the in- coming connections between hosts in an Ethernet network. It only works for clients in the local network, because ARP balancing spreads the load by varying ARP replies based on the source MAC address of the host send- ing the query. Therefore it cannot balance traffic that crosses a router, because the router itself will always be balanced to the same virtual host.
IP balancing is not dependent on ARP and therefore also works for traffic that comes over a router. This method should work in all environments and can also provide more fine grained load balancing than ARP balancing. The downside of IP balancing is that it requires the traffic that is des- tined towards the load balanced IP addresses to be received by all carp hosts. While this is always the case when connected to a hub, it has to play some tricks in switched networks, which will result in a higher net- work load.
A rule of thumb might be to use ARP balancing if there are many hosts on the same network segment and to use IP balancing for all other cases.
To configure load balancing one has to specify multiple carp nodes using the carpnodes option. Each node in a load balancing cluster is repre- sented by at least one “vhid:advskew” pair in a comma separated list. carp tries to distribute the incoming network load over all configured carpnodes. The following example creates a load balancing group consist- ing of three nodes, using vhids 3, 4 and 6:
# ifconfig carp0 carpnodes 3:0,4:0,6:100
The advskew value of the last node is set to 100, so that this node is designated to the BACKUP state. It will only become MASTER if all nodes with a lower advskew value have failed. By varying this value throughout the machines in the cluster it is possible to decide which share of the network load each node receives. Therefore, all carp interfaces in the cluster are configured identically, except for a different advskew value within the carpnodes specification.
See the EXAMPLES section for a practical example of load balancing.
ARP BALANCING For ARP balancing, one has to configure multiple carpnodes and choose the balancing mode arp.
Once an ARP request is received, the CARP protocol will use a hashing function against the source MAC address in the ARP request to determine which carpnode the request belongs to. If the corresponding carpnode is in master state, the ARP request will be answered, otherwise it will be ignored.
The ARP load balancing has some limitations. Firstly, ARP balancing only works on the local network segment. It cannot balance traffic that crosses a router, because the router itself will always be balanced to the same carpnode. Secondly, ARP load balancing can lead to asymmetric routing of incoming and outgoing traffic, thus combining it with pfsync(4) requires special care, because this can create a race condition between balanced routers and the host they are serving. ARP balancing can be safely used with pfsync if the pf(4) ruleset translates the source address to an unshared address on the outgoing interface using a NAT rule. This requires multiple CARP groups with different IP addresses on the outgoing interface, configured so that each host is the master of one group.
ARP balancing also works for IPv6, but instead of ARP the Neighbor Dis- covery Protocol (NDP) is used.
IP BALANCING IP load balancing works by utilizing the network itself to distribute in- coming traffic to all carp nodes in the cluster. Each packet is filtered on the incoming carp interface so that only one node in the cluster ac- cepts the packet. All the other nodes will just silently drop it. The filtering function uses a hash over the source and destination address of the IPv4 or IPv6 packet and compares the result against the state of the carpnode.
IP balancing is activated by setting the balancing mode to ip. This is the recommended default setting. In this mode, carp uses a multicast MAC address, so that a switch sends incoming traffic towards all nodes.
However, there are a few OS and routers that do not accept a multicast MAC address being mapped to a unicast IP. This can be resolved by using one of the following unicast options. For scenarios where a hub is used it is not necessary to use a multicast MAC and it is safe to use the ip- unicast mode. Manageable switches can usually be tricked into forwarding unicast traffic to all cluster nodes ports by configuring them into some sort of monitoring mode. If this is not possible, using the ip-stealth mode is another option, which should work on most switches. In this mode carp never sends packets with its virtual MAC address as source. Stealth mode prevents a switch from learning the virtual MAC address, so that it has to flood the traffic to all its ports. Please note that activating stealth mode on a carp interface that has already been running might not work instantly. As a workaround the VHID of the first carpnode can be changed to a previously unused one, or just wait until the MAC table en- try in the switch times out. Some Layer-3 switches do port learning based on ARP packets. Therefore the stealth mode cannot hide the virtual MAC address from these kind of devices.
If IP balancing is being used on a firewall, it is recommended to config- ure the carpnodes in a symmetrical manner. This is achieved by simply using the same carpnodes list on all sides of the firewall. This ensures that packets of one connection will pass in and out on the same host and are not routed asymmetrically.
EXAMPLES For firewalls and routers with multiple interfaces, it is desirable to failover all of the carp interfaces together, when one of the physical interfaces goes down. This is achieved by the preempt option. Enable it on both host A and B:
# sysctl net.inet.carp.preempt=1
Assume that host A is the preferred master and 192.168.1.x/24 is config- ured on one physical interface and 192.168.2.y/24 on another. This is the setup for host A:
# ifconfig carp0 192.168.1.1 vhid 1 # ifconfig carp1 192.168.2.1 vhid 2
The setup for host B is identical, but it has a higher advskew:
# ifconfig carp0 192.168.1.1 vhid 1 advskew 100 # ifconfig carp1 192.168.2.1 vhid 2 advskew 100
Because of the preempt option, when one of the physical interfaces of host A fails, advskew is adjusted to 240 on all its carp interfaces. This will cause host B to preempt on both interfaces instead of just the failed one.
LOAD BALANCING In order to set up a load balanced virtual host, it is necessary to con- figure one carpnodes entry for each physical host. In the following ex- ample, two physical hosts are configured to provide balancing and failover for the IP address 192.168.1.10.
First the carp interface on Host A is configured. The advskew of 100 on the second carpnode entry means that its advertisements will be sent out slightly less frequently and will therefore become the designated backup.
# ifconfig carp0 192.168.1.10 carpnodes 1:0,2:100 balancing ip
The configuration for host B is identical, except the skew is on the carpnode entry with virtual host 1 rather than virtual host 2.
# ifconfig carp0 192.168.1.10 carpnodes 1:100,2:0 balancing ip
If ARP balancing or a different mode of IP balancing is desired the balancing mode can be adjusted accordingly.
SEE ALSO sysctl(3), inet(4), pfsync(4), hostname.if(5), ifconfig(8), ifstated(8), netstart(8), sysctl(8)