1.\" $OpenBSD: carp.4,v 1.16 2004/12/07 23:41:35 jmc Exp $ 2.\" 3.\" Copyright (c) 2003, Ryan McBride. All rights reserved. 4.\" Copyright (c) 2011, Gleb Smirnoff <glebius@FreeBSD.org> 5.\" 6.\" Redistribution and use in source and binary forms, with or without 7.\" modification, are permitted provided that the following conditions 8.\" are met: 9.\" 1. Redistributions of source code must retain the above copyright 10.\" notice, this list of conditions and the following disclaimer. 11.\" 2. Redistributions in binary form must reproduce the above copyright 12.\" notice, this list of conditions and the following disclaimer in the 13.\" documentation and/or other materials provided with the distribution. 14.\" 15.\" THIS SOFTWARE IS PROVIDED BY THE PROJECT AND CONTRIBUTORS ``AS IS'' AND 16.\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 17.\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE 18.\" ARE DISCLAIMED. IN NO EVENT SHALL THE PROJECT OR CONTRIBUTORS BE LIABLE 19.\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 20.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS 21.\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) 22.\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT 23.\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY 24.\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF 25.\" SUCH DAMAGE. 26.\" 27.\" $FreeBSD$ 28.\" 29.Dd February 21, 2013 30.Dt CARP 4 31.Os 32.Sh NAME 33.Nm carp 34.Nd Common Address Redundancy Protocol 35.Sh SYNOPSIS 36.Cd "device carp" 37.Sh DESCRIPTION 38The CARP allows multiple hosts on the same local network to share a set of 39IPv4 and/or IPv6 addresses. 40Its primary purpose is to ensure that these 41addresses are always available. 42.Pp 43To use 44.Nm , 45the administrator needs to configure at a minimum a common virtual host ID 46(vhid), and attach at least one IP address to this vhid on each machine which 47is to take part in the virtual group. 48Additional parameters can also be set on a per-vhid basis: 49.Cm advbase 50and 51.Cm advskew , 52which are used to control how frequently the host sends advertisements when it 53is the master for a virtual host, and 54.Cm pass 55which is used to authenticate 56.Nm 57advertisements. 58The 59.Cm advbase 60parameter stands for 61.Dq "advertisement base" . 62It is measured in seconds and specifies the base of the advertisement interval. 63The 64.Cm advskew 65parameter stands for 66.Dq "advertisement skew" . 67It is measured in 1/256 of seconds. 68It is added to the base advertisement interval to make one host advertise 69a bit slower that the other does. 70Both 71.Cm advbase 72and 73.Cm advskew 74are put inside CARP advertisements. 75These values can be configured using 76.Xr ifconfig 8 , 77or through the 78.Dv SIOCSVH 79.Xr ioctl 2 . 80.Pp 81CARP virtual hosts can be configured on multicast-capable interfaces: Ethernet, 82layer 2 VLAN, FDDI and Token Ring. 83An arbitrary number of virtual host IDs can be configured on an interface. 84An arbitrary number of IPv4 or IPv6 addresses can be attached to a particular 85vhid. 86It is important that all hosts participating in a vhid have the same list 87of prefixes configured on the vhid, since all prefixes are included in the 88cryptographic checksum supplied in each advertisement. 89Multiple vhids running on one interface participate in master/backup 90elections independently. 91.Pp 92Additionally, there are a number of global parameters which can be set using 93.Xr sysctl 8 : 94.Bl -tag -width ".Va net.inet.carp.ifdown_demotion_factor" 95.It Va net.inet.carp.allow 96Accept incoming 97.Nm 98packets. 99Enabled by default. 100.It Va net.inet.carp.preempt 101Allow virtual hosts to preempt each other. 102When enabled, a vhid in a backup state would preempt a master that 103is announcing itself with a lower advskew. 104Disabled by default. 105.It Va net.inet.carp.log 106Determines what events relating to 107.Nm 108vhids are logged. 109A value of 0 disables any logging. 110A value of 1 enables logging state changes of 111.Nm 112vhids. 113Values above 1 enable logging of bad 114.Nm 115packets. 116The default value is 1. 117.It Va net.inet.carp.demotion 118This value shows current level of CARP demotion. 119The value is added to the actual advskew sent in announcements for 120all vhids. 121At normal system operation the demotion factor is zero. 122However, problematic conditions raise its level: when 123.Nm 124experiences problem with sending announcements, when an interface 125running a vhid goes down, or while the 126.Xr pfsync 4 127interface is not synchronized. 128The demotion factor can be adjusted writing to the sysctl oid. 129The signed value supplied to the 130.Xr sysctl 8 131command is added to current demotion factor. 132This allows to control 133.Nm 134behaviour depending on some external conditions, for example on the status 135of some daemon utility. 136.It Va net.inet.carp.ifdown_demotion_factor 137This value is added to 138.Va net.inet.carp.demotion 139when an interface running a vhid goes down. 140The default value is 240 (the maximum advskew value). 141.It Va net.inet.carp.senderr_demotion_factor 142This value is added to 143.Va net.inet.carp.demotion 144when 145.Nm 146experiences errors sending its announcements. 147The default value is 240 (the maximum advskew value). 148.El 149.\".Sh ARP level load balancing 150.\"A 151.\".Nm 152.\"interface has limited abilities for load balancing incoming connections 153.\"between hosts in an Ethernet network. 154.\"For load-balancing operation, one needs several CARP interfaces that 155.\"are configured to the same IP address, but to a different vhids. 156.\"Once an ARP request is received, the CARP protocol will use a hashing 157.\"function against the source IP address in the ARP request to determine 158.\"which vhid the request will be assigned to. 159.\"If the corresponding CARP interface is the current 160.\"master interface, a reply will 161.\"be sent to the ARP request; 162.\"otherwise it will be ignored. 163.\"See the 164.\".Sx EXAMPLES 165.\"section for a practical example of load balancing. 166.\".Pp 167.\"The ARP load balancing implemented in 168.\".Nm 169.\"has some limitations. 170.\"First, ARP balancing only works on the local network segment. 171.\"It cannot balance traffic that crosses a router, because the 172.\"router itself will always be balanced to the same virtual host. 173.\"Second, ARP load balancing can lead to asymmetric routing 174.\"of incoming and outgoing traffic, and thus combining it with 175.\".Xr pfsync 4 176.\"is dangerous, because this creates a race condition between 177.\"balanced routers and a host they are serving. 178.\"Imagine an incoming packet creating state on the first router, being 179.\"forwarded to its destination, and the destination replying faster 180.\"than the state information is packed and synced with the second router. 181.\"If the reply would be load balanced to second router, it will be 182.\"dropped since the second router has not yet received information about 183.\"the connection state. 184.Sh STATE CHANGE NOTIFICATIONS 185Sometimes it is useful to get notified about 186.Nm 187status change events. 188This can be accomplished by using 189.Xr devd 8 190hooks. 191Master/slave events are signalled under system 192.Dv CARP . 193The subsystem specifies the vhid and name of the interface where 194the master/slave event occurred. 195The type of the message displays the new state of the vhid. 196Please see 197.Xr devd.conf 5 198and the 199.Sx EXAMPLES 200section for more information. 201.Sh EXAMPLES 202For firewalls and routers with multiple interfaces, it is desirable to 203failover all of the addresses running 204.Nm 205together, when one of the physical interfaces goes down. 206This is achieved by the use of the preempt option. 207Enable it on both hosts A and B: 208.Pp 209.Dl sysctl net.inet.carp.preempt=1 210.Pp 211Assume that host A is the preferred master and we are running the 212192.168.1.0/24 prefix on em0 and 192.168.2.0/24 on em1. 213This is the setup for host A (advskew is above 0 so it could be overwritten 214in the emergency situation from the other host): 215.Bd -literal -offset indent 216ifconfig em0 vhid 1 advskew 100 pass mekmitasdigoat 192.168.1.1/24 217ifconfig em1 vhid 2 advskew 100 pass mekmitasdigoat 192.168.2.1/24 218.Ed 219.Pp 220The setup for host B is identical, but it has a higher 221.Cm advskew : 222.Bd -literal -offset indent 223ifconfig em0 vhid 1 advskew 200 pass mekmitasdigoat 192.168.1.1/24 224ifconfig em1 vhid 2 advskew 200 pass mekmitasdigoat 192.168.2.1/24 225.Ed 226.Pp 227When one of the physical interfaces of host A fails, 228.Cm advskew 229is demoted to a configured value on all its 230.Nm 231vhids. 232Due to the preempt option, host B would start announcing itself, and thus 233preempt host A on both interfaces instead of just the failed one. 234.\".Pp 235.\"In order to set up an ARP balanced virtual host, it is necessary to configure 236.\"one virtual host for each physical host which would respond to ARP requests 237.\"and thus handle the traffic. 238.\"In the following example, two virtual hosts are configured on two hosts to 239.\"provide balancing and failover for the IP address 192.168.1.10. 240.\".Pp 241.\"First the 242.\".Nm 243.\"interfaces on host A are configured. 244.\"The 245.\".Cm advskew 246.\"of 100 on the second virtual host means that its advertisements will be sent 247.\"out slightly less frequently. 248.\".Bd -literal -offset indent 249.\"ifconfig carp0 create 250.\"ifconfig carp0 vhid 1 advskew 100 pass mekmitasdigoat 192.168.1.10/24 251.\"ifconfig carp1 create 252.\"ifconfig carp1 vhid 2 advskew 200 pass mekmitasdigoat 192.168.1.10/24 253.\".Ed 254.\".Pp 255.\"The configuration for host B is identical, except the 256.\".Cm advskew 257.\"is on virtual host 1 rather than virtual host 2. 258.\".Bd -literal -offset indent 259.\"ifconfig carp0 create 260.\"ifconfig carp0 vhid 1 advskew 200 pass mekmitasdigoat 192.168.1.10/24 261.\"ifconfig carp1 create 262.\"ifconfig carp1 vhid 2 advskew 100 pass mekmitasdigoat 192.168.1.10/24 263.\".Ed 264.\".Pp 265.\"Finally, the ARP balancing feature must be enabled on both hosts: 266.\".Pp 267.\".Dl sysctl net.inet.carp.arpbalance=1 268.\".Pp 269.\"When the hosts receive an ARP request for 192.168.1.10, the source IP address 270.\"of the request is used to compute which virtual host should answer the request. 271.\"The host which is master of the selected virtual host will reply to the 272.\"request, the other(s) will ignore it. 273.\".Pp 274.\"This way, locally connected systems will receive different ARP replies and 275.\"subsequent IP traffic will be balanced among the hosts. 276.\"If one of the hosts fails, the other will take over the virtual MAC address, 277.\"and begin answering ARP requests on its behalf. 278.Pp 279Processing of 280.Nm 281status change events can be set up by using the following devd.conf rule: 282.Bd -literal -offset indent 283notify 0 { 284 match "system" "CARP"; 285 match "subsystem" "[0-9]+@[0-9a-z]+"; 286 match "type" "(MASTER|BACKUP)"; 287 action "/root/carpcontrol.sh $subsystem $type"; 288}; 289.Ed 290.Pp 291To see 292.Nm 293packets decoded in 294.Xr tcpdump 1 295output, one needs to specify 296.Fl T Ar carp 297option, otherwise 298.Xr tcpdump 1 299tries to interpret them as VRRP packets: 300.Bd -literal -offset indent 301tcpdump -npi vlan0 -T carp 302.Ed 303.Sh SEE ALSO 304.Xr tcpdump 1 , 305.Xr inet 4 , 306.Xr pfsync 4 , 307.Xr devd.conf 5 , 308.Xr rc.conf 5 , 309.Xr ifconfig 8 , 310.Xr sysctl 8 311.Sh HISTORY 312The 313.Nm 314device first appeared in 315.Ox 3.5 . 316The 317.Nm 318device was imported into 319.Fx 5.4 . 320In 321.Fx 10.0 , 322.Nm 323was significantly rewritten, and is no longer a pseudo-interface. 324