1.\" $OpenBSD: carp.4,v 1.16 2004/12/07 23:41:35 jmc Exp $ 2.\" 3.\" Copyright (c) 2003, Ryan McBride. All rights reserved. 4.\" Copyright (c) 2011, Gleb Smirnoff <glebius@FreeBSD.org> 5.\" 6.\" Redistribution and use in source and binary forms, with or without 7.\" modification, are permitted provided that the following conditions 8.\" are met: 9.\" 1. Redistributions of source code must retain the above copyright 10.\" notice, this list of conditions and the following disclaimer. 11.\" 2. Redistributions in binary form must reproduce the above copyright 12.\" notice, this list of conditions and the following disclaimer in the 13.\" documentation and/or other materials provided with the distribution. 14.\" 15.\" THIS SOFTWARE IS PROVIDED BY THE PROJECT AND CONTRIBUTORS ``AS IS'' AND 16.\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 17.\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE 18.\" ARE DISCLAIMED. IN NO EVENT SHALL THE PROJECT OR CONTRIBUTORS BE LIABLE 19.\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 20.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS 21.\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) 22.\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT 23.\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY 24.\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF 25.\" SUCH DAMAGE. 26.\" 27.\" $FreeBSD$ 28.\" 29.Dd May 7, 2018 30.Dt CARP 4 31.Os 32.Sh NAME 33.Nm carp 34.Nd Common Address Redundancy Protocol 35.Sh SYNOPSIS 36.Cd "device carp" 37.Sh DESCRIPTION 38The CARP allows multiple hosts on the same local network to share a set of 39IPv4 and/or IPv6 addresses. 40Its primary purpose is to ensure that these 41addresses are always available. 42.Pp 43To use 44.Nm , 45the administrator needs to configure at a minimum a common virtual host ID 46(vhid), and attach at least one IP address to this vhid on each machine which 47is to take part in the virtual group. 48Additional parameters can also be set on a per-vhid basis: 49.Cm advbase 50and 51.Cm advskew , 52which are used to control how frequently the host sends advertisements when it 53is the master for a virtual host, and 54.Cm pass 55which is used to authenticate 56.Nm 57advertisements. 58The 59.Cm advbase 60parameter stands for 61.Dq "advertisement base" . 62It is measured in seconds and specifies the base of the advertisement interval. 63The 64.Cm advskew 65parameter stands for 66.Dq "advertisement skew" . 67It is measured in 1/256 of seconds. 68It is added to the base advertisement interval to make one host advertise 69a bit slower that the other does. 70Both 71.Cm advbase 72and 73.Cm advskew 74are put inside CARP advertisements. 75These values can be configured using 76.Xr ifconfig 8 , 77or through the 78.Dv SIOCSVH 79.Xr ioctl 2 . 80.Pp 81CARP virtual hosts can be configured on multicast-capable interfaces: Ethernet, 82layer 2 VLAN, FDDI and Token Ring. 83An arbitrary number of virtual host IDs can be configured on an interface. 84An arbitrary number of IPv4 or IPv6 addresses can be attached to a particular 85vhid. 86It is important that all hosts participating in a vhid have the same list 87of prefixes configured on the vhid, since all prefixes are included in the 88cryptographic checksum supplied in each advertisement. 89Multiple vhids running on one interface participate in master/backup 90elections independently. 91.Pp 92Additionally, there are a number of global parameters which can be set using 93.Xr sysctl 8 : 94.Bl -tag -width ".Va net.inet.carp.ifdown_demotion_factor" 95.It Va net.inet.carp.allow 96Allow 97.Nm 98operation. 99When disabled, virtual hosts remain in initial state, neither sending nor 100receiving announcements or traffic. 101Enabled by default. 102.It Va net.inet.carp.preempt 103Allow virtual hosts to preempt each other. 104When enabled, a vhid in a backup state would preempt a master that 105is announcing itself with a lower advskew. 106Disabled by default. 107.It Va net.inet.carp.log 108Determines what events relating to 109.Nm 110vhids are logged. 111A value of 0 disables any logging. 112A value of 1 enables logging state changes of 113.Nm 114vhids. 115Values above 1 enable logging of bad 116.Nm 117packets. 118The default value is 1. 119.It Va net.inet.carp.demotion 120This value shows current level of CARP demotion. 121The value is added to the actual advskew sent in announcements for 122all vhids. 123At normal system operation the demotion factor is zero. 124However, problematic conditions raise its level: when 125.Nm 126experiences problem with sending announcements, when an interface 127running a vhid goes down, or while the 128.Xr pfsync 4 129interface is not synchronized. 130The demotion factor can be adjusted writing to the sysctl oid. 131The signed value supplied to the 132.Xr sysctl 8 133command is added to current demotion factor. 134This allows to control 135.Nm 136behaviour depending on some external conditions, for example on the status 137of some daemon utility. 138.It Va net.inet.carp.ifdown_demotion_factor 139This value is added to 140.Va net.inet.carp.demotion 141when an interface running a vhid goes down. 142The default value is 240 (the maximum advskew value). 143.It Va net.inet.carp.senderr_demotion_factor 144This value is added to 145.Va net.inet.carp.demotion 146when 147.Nm 148experiences errors sending its announcements. 149The default value is 240 (the maximum advskew value). 150.El 151.\".Sh ARP level load balancing 152.\"A 153.\".Nm 154.\"interface has limited abilities for load balancing incoming connections 155.\"between hosts in an Ethernet network. 156.\"For load-balancing operation, one needs several CARP interfaces that 157.\"are configured to the same IP address, but to a different vhids. 158.\"Once an ARP request is received, the CARP protocol will use a hashing 159.\"function against the source IP address in the ARP request to determine 160.\"which vhid the request will be assigned to. 161.\"If the corresponding CARP interface is the current 162.\"master interface, a reply will 163.\"be sent to the ARP request; 164.\"otherwise it will be ignored. 165.\"See the 166.\".Sx EXAMPLES 167.\"section for a practical example of load balancing. 168.\".Pp 169.\"The ARP load balancing implemented in 170.\".Nm 171.\"has some limitations. 172.\"First, ARP balancing only works on the local network segment. 173.\"It cannot balance traffic that crosses a router, because the 174.\"router itself will always be balanced to the same virtual host. 175.\"Second, ARP load balancing can lead to asymmetric routing 176.\"of incoming and outgoing traffic, and thus combining it with 177.\".Xr pfsync 4 178.\"is dangerous, because this creates a race condition between 179.\"balanced routers and a host they are serving. 180.\"Imagine an incoming packet creating state on the first router, being 181.\"forwarded to its destination, and the destination replying faster 182.\"than the state information is packed and synced with the second router. 183.\"If the reply would be load balanced to second router, it will be 184.\"dropped since the second router has not yet received information about 185.\"the connection state. 186.Sh STATE CHANGE NOTIFICATIONS 187Sometimes it is useful to get notified about 188.Nm 189status change events. 190This can be accomplished by using 191.Xr devd 8 192hooks. 193Master/slave events are signalled under system 194.Dv CARP . 195The subsystem specifies the vhid and name of the interface where 196the master/slave event occurred. 197The type of the message displays the new state of the vhid. 198Please see 199.Xr devd.conf 5 200and the 201.Sx EXAMPLES 202section for more information. 203.Sh EXAMPLES 204For firewalls and routers with multiple interfaces, it is desirable to 205failover all of the addresses running 206.Nm 207together, when one of the physical interfaces goes down. 208This is achieved by the use of the preempt option. 209Enable it on both hosts A and B: 210.Pp 211.Dl sysctl net.inet.carp.preempt=1 212.Pp 213Assume that host A is the preferred master and we are running the 214192.168.1.0/24 prefix on em0 and 192.168.2.0/24 on em1. 215This is the setup for host A (advskew is above 0 so it could be overwritten 216in the emergency situation from the other host): 217.Bd -literal -offset indent 218ifconfig em0 vhid 1 advskew 100 pass mekmitasdigoat 192.168.1.1/24 219ifconfig em1 vhid 2 advskew 100 pass mekmitasdigoat 192.168.2.1/24 220.Ed 221.Pp 222The setup for host B is identical, but it has a higher 223.Cm advskew : 224.Bd -literal -offset indent 225ifconfig em0 vhid 1 advskew 200 pass mekmitasdigoat 192.168.1.1/24 226ifconfig em1 vhid 2 advskew 200 pass mekmitasdigoat 192.168.2.1/24 227.Ed 228.Pp 229When one of the physical interfaces of host A fails, 230.Cm advskew 231is demoted to a configured value on all its 232.Nm 233vhids. 234Due to the preempt option, host B would start announcing itself, and thus 235preempt host A on both interfaces instead of just the failed one. 236.\".Pp 237.\"In order to set up an ARP balanced virtual host, it is necessary to configure 238.\"one virtual host for each physical host which would respond to ARP requests 239.\"and thus handle the traffic. 240.\"In the following example, two virtual hosts are configured on two hosts to 241.\"provide balancing and failover for the IP address 192.168.1.10. 242.\".Pp 243.\"First the 244.\".Nm 245.\"interfaces on host A are configured. 246.\"The 247.\".Cm advskew 248.\"of 100 on the second virtual host means that its advertisements will be sent 249.\"out slightly less frequently. 250.\".Bd -literal -offset indent 251.\"ifconfig carp0 create 252.\"ifconfig carp0 vhid 1 advskew 100 pass mekmitasdigoat 192.168.1.10/24 253.\"ifconfig carp1 create 254.\"ifconfig carp1 vhid 2 advskew 200 pass mekmitasdigoat 192.168.1.10/24 255.\".Ed 256.\".Pp 257.\"The configuration for host B is identical, except the 258.\".Cm advskew 259.\"is on virtual host 1 rather than virtual host 2. 260.\".Bd -literal -offset indent 261.\"ifconfig carp0 create 262.\"ifconfig carp0 vhid 1 advskew 200 pass mekmitasdigoat 192.168.1.10/24 263.\"ifconfig carp1 create 264.\"ifconfig carp1 vhid 2 advskew 100 pass mekmitasdigoat 192.168.1.10/24 265.\".Ed 266.\".Pp 267.\"Finally, the ARP balancing feature must be enabled on both hosts: 268.\".Pp 269.\".Dl sysctl net.inet.carp.arpbalance=1 270.\".Pp 271.\"When the hosts receive an ARP request for 192.168.1.10, the source IP address 272.\"of the request is used to compute which virtual host should answer the request. 273.\"The host which is master of the selected virtual host will reply to the 274.\"request, the other(s) will ignore it. 275.\".Pp 276.\"This way, locally connected systems will receive different ARP replies and 277.\"subsequent IP traffic will be balanced among the hosts. 278.\"If one of the hosts fails, the other will take over the virtual MAC address, 279.\"and begin answering ARP requests on its behalf. 280.Pp 281Processing of 282.Nm 283status change events can be set up by using the following devd.conf rule: 284.Bd -literal -offset indent 285notify 0 { 286 match "system" "CARP"; 287 match "subsystem" "[0-9]+@[0-9a-z]+"; 288 match "type" "(MASTER|BACKUP)"; 289 action "/root/carpcontrol.sh $subsystem $type"; 290}; 291.Ed 292.Pp 293To see 294.Nm 295packets decoded in 296.Xr tcpdump 1 297output, one needs to specify 298.Fl T Ar carp 299option, otherwise 300.Xr tcpdump 1 301tries to interpret them as VRRP packets: 302.Bd -literal -offset indent 303tcpdump -npi vlan0 -T carp 304.Ed 305.Sh SEE ALSO 306.Xr tcpdump 1 , 307.Xr inet 4 , 308.Xr pfsync 4 , 309.Xr devd.conf 5 , 310.Xr rc.conf 5 , 311.Xr ifconfig 8 , 312.Xr sysctl 8 313.Sh HISTORY 314The 315.Nm 316device first appeared in 317.Ox 3.5 . 318The 319.Nm 320device was imported into 321.Fx 5.4 . 322In 323.Fx 10.0 , 324.Nm 325was significantly rewritten, and is no longer a pseudo-interface. 326