1.\" Copyright (c) 2007 Matthew Jacob 2.\" All rights reserved. 3.\" 4.\" Redistribution and use in source and binary forms, with or without 5.\" modification, are permitted provided that the following conditions 6.\" are met: 7.\" 1. Redistributions of source code must retain the above copyright 8.\" notice, this list of conditions and the following disclaimer. 9.\" 2. Redistributions in binary form must reproduce the above copyright 10.\" notice, this list of conditions and the following disclaimer in the 11.\" documentation and/or other materials provided with the distribution. 12.\" 13.\" THIS SOFTWARE IS PROVIDED BY THE AUTHORS AND CONTRIBUTORS ``AS IS'' AND 14.\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 15.\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE 16.\" ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHORS OR CONTRIBUTORS BE LIABLE 17.\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 18.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS 19.\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) 20.\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT 21.\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY 22.\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF 23.\" SUCH DAMAGE. 24.\" 25.\" $FreeBSD$ 26.\" 27.Dd March 17, 2022 28.Dt GMULTIPATH 8 29.Os 30.Sh NAME 31.Nm gmultipath 32.Nd "disk multipath control utility" 33.Sh SYNOPSIS 34.Nm 35.Cm create 36.Op Fl ARv 37.Ar name 38.Ar prov ... 39.Nm 40.Cm label 41.Op Fl ARv 42.Ar name 43.Ar prov ... 44.Nm 45.Cm configure 46.Op Fl APRv 47.Ar name 48.Nm 49.Cm add 50.Op Fl v 51.Ar name prov 52.Nm 53.Cm remove 54.Op Fl v 55.Ar name prov 56.Nm 57.Cm fail 58.Op Fl v 59.Ar name prov 60.Nm 61.Cm restore 62.Op Fl v 63.Ar name prov 64.Nm 65.Cm rotate 66.Op Fl v 67.Ar name 68.Nm 69.Cm prefer 70.Op Fl v 71.Ar name 72.Ar prov 73.Nm 74.Cm getactive 75.Op Fl v 76.Ar name 77.Nm 78.Cm destroy 79.Op Fl v 80.Ar name 81.Nm 82.Cm stop 83.Op Fl v 84.Ar name 85.Nm 86.Cm clear 87.Op Fl v 88.Ar prov ... 89.Nm 90.Cm list 91.Nm 92.Cm status 93.Nm 94.Cm load 95.Nm 96.Cm unload 97.Sh DESCRIPTION 98The 99.Nm 100utility is used for device multipath configuration. 101.Pp 102The multipath device can be configured using two different methods: 103.Dq manual 104or 105.Dq automatic . 106When using the 107.Dq manual 108method, no metadata are stored on the devices, so the multipath 109device has to be configured by hand every time it is needed. 110Additional device paths also will not be detected automatically. 111The 112.Dq automatic 113method uses on-disk metadata to detect device and all its paths. 114Metadata use the last sector of the underlying disk device and 115include device name and UUID. 116The UUID guarantees uniqueness in a shared storage environment 117but is in general too cumbersome to use. 118The name is what is exported via the device interface. 119.Pp 120The first argument to 121.Nm 122indicates an action to be performed: 123.Bl -tag -width ".Cm destroy" 124.It Cm create 125Create multipath device with 126.Dq manual 127method without writing any on-disk metadata. 128It is up to administrator, how to properly identify device paths. 129Kernel will only check that all given providers have same media and 130sector sizes. 131.Pp 132.Fl A 133option enables Active/Active mode, 134.Fl R 135option enables Active/Read mode, otherwise Active/Passive mode is used 136by default. 137.It Cm label 138Create multipath device with 139.Dq automatic 140method. 141Label the first given provider with on-disk metadata using the specified 142.Ar name . 143The rest of given providers will be retasted to detect these metadata. 144It reliably protects against specifying unrelated providers. 145Providers with no matching metadata detected will not be added to the device. 146.Pp 147.Fl A 148option enables Active/Active mode, 149.Fl R 150option enables Active/Read mode, otherwise Active/Passive mode is used 151by default. 152.It Cm configure 153Configure the given multipath device. 154.Pp 155.Fl A 156option enables Active/Active mode, 157.Fl P 158option enables Active/Passive mode, 159.Fl R 160option enables Active/Read mode. 161.It Cm add 162Add the given provider as a path to the given multipath device. 163Should normally be used only for devices created with 164.Dq manual 165method, unless you know what you are doing (you are sure that it is another 166device path, but tasting its metadata in regular 167.Dq automatic 168way is not possible). 169.It Cm remove 170Remove the given provider as a path from the given multipath device. 171If the last path removed, the multipath device will be destroyed. 172.It Cm fail 173Mark specified provider as a path of the specified multipath device as failed. 174If there are other paths present, new requests will be forwarded there. 175.It Cm restore 176Mark specified provider as a path of the specified multipath device as 177operational, allowing it to handle requests. 178.It Cm rotate 179Change the active provider/path to the next available provider in Active/Passive mode. 180.It Cm prefer 181Change the active provider/path to the specified provider in Active/Passive mode. 182.It Cm getactive 183Get the currently active provider(s)/path(s). 184.It Cm destroy 185Destroy the given multipath device clearing metadata. 186.It Cm stop 187Stop the given multipath device without clearing metadata. 188.It Cm clear 189Clear metadata on the given provider. 190.It Cm list 191See 192.Xr geom 8 . 193.It Cm status 194See 195.Xr geom 8 . 196.It Cm load 197See 198.Xr geom 8 . 199.It Cm unload 200See 201.Xr geom 8 . 202.El 203.Sh SYSCTL VARIABLES 204The following 205.Xr sysctl 8 206variable can be used to control the behavior of the 207.Nm MULTIPATH 208GEOM class. 209.Bl -tag -width indent 210.It Va kern.geom.multipath.debug : No 0 211Debug level of the 212.Nm MULTIPATH 213GEOM class. 214This can be set to 0 (default) or 1 to disable or enable various 215forms of chattiness. 216.It Va kern.geom.multipath.exclusive : No 1 217Open underlying providers exclusively, preventing individual paths access. 218.El 219.Sh EXIT STATUS 220Exit status is 0 on success, and 1 if the command fails. 221.Sh MULTIPATH ARCHITECTURE 222This is a multiple path architecture with no device knowledge or 223presumptions other than size matching built in. 224Therefore the user must exercise some care 225in selecting providers that do indeed represent multiple paths to the 226same underlying disk device. 227The reason for this is that there are several 228criteria across multiple underlying transport types that can 229.Ar indicate 230identity, but in all respects such identity can rarely be considered 231.Ar definitive . 232.Pp 233For example, if you use the World Wide Port Name of a Fibre Channel 234disk object you might believe that two disks that have the same WWPN 235on different paths (or even disjoint fabrics) might be considered 236the same disk. 237Nearly always this would be a safe assumption, until 238you realize that a WWPN, like an Ethernet MAC address, is a soft 239programmable entity, and that a misconfigured Director Class switch 240could lead you to believe incorrectly that you have found multiple 241paths to the same device. 242This is an extreme and theoretical case, but 243it is possible enough to indicate that the policy for deciding which 244of multiple pathnames refer to the same device should be left to the 245system operator who will use tools and knowledge of their own storage 246subsystem to make the correct configuration selection. 247.Pp 248There are Active/Passive, Active/Read and Active/Active operation modes 249supported. 250In Active/Passive mode only one path has I/O moving on it 251at any point in time. 252This I/O continues until an I/O is returned with 253a generic I/O error or a "Nonexistent Device" error. 254When this occurs, that path is marked FAIL, the next path 255in a list is selected as active and the failed I/O reissued. 256In Active/Active mode all paths not marked FAIL may handle I/O at the same time. 257Requests are distributed between paths to equalize load. 258For capable devices it allows the utilisation of the bandwidth available on all paths. 259In Active/Read mode all paths not marked FAIL may handle reads at the same time, 260but unlike in Active/Active mode only one path handles write requests at any 261point in time; closely following the original write request order if the layer 262above needs it for data consistency (not waiting for requisite write completion 263before sending dependent write). 264.Pp 265When new devices are added to the system the 266.Nm MULTIPATH 267GEOM class is given an opportunity to taste these new devices. 268If a new 269device has a 270.Nm MULTIPATH 271on-disk metadata label, the device is either used to create a new 272.Nm MULTIPATH 273GEOM, or added to the list of paths for an existing 274.Nm MULTIPATH 275GEOM. 276.Pp 277It is this mechanism that works reasonably with 278.Xr isp 4 279and 280.Xr mpt 4 281based Fibre Channel disk devices. 282For these devices, when a device disappears 283(due to e.g., a cable pull or power failure to a switch), the device is 284proactively marked as gone and I/O to it failed. 285This causes the 286.Nm MULTIPATH 287failure event just described. 288.Pp 289When Fibre Channel events inform either 290.Xr isp 4 291or 292.Xr mpt 4 293host bus adapters that new devices may have arrived (e.g., the arrival 294of an RSCN event from the Fabric Domain Controller), they can cause 295a rescan to occur and cause the attachment and configuration of any 296(now) new devices to occur, causing the taste event described above. 297.Pp 298This means that this multipath architecture is not a one-shot path 299failover, but can be considered to be steady state as long as failed 300paths are repaired (automatically or otherwise). 301.Pp 302Automatic rescanning is not a requirement. 303Nor is Fibre Channel. 304The 305same failover mechanisms work equally well for traditional "Parallel" 306SCSI but may require manual intervention with 307.Xr camcontrol 8 308to cause the reattachment of repaired device links. 309.Sh EXAMPLES 310The following example shows how to use 311.Xr camcontrol 8 312to find possible multiple path devices and to create a 313.Nm MULTIPATH 314GEOM class for them. 315.Bd -literal -offset indent 316mysys# camcontrol devlist 317<ECNCTX @WESTVILLE > at scbus0 target 0 lun 0 (da0,pass0) 318<ECNCTX @WESTVILLE > at scbus0 target 0 lun 1 (da1,pass1) 319<ECNCTX @WESTVILLE > at scbus1 target 0 lun 0 (da2,pass2) 320<ECNCTX @WESTVILLE > at scbus1 target 0 lun 1 (da3,pass3) 321mysys# camcontrol inquiry da0 -S 322ECNTX0LUN000000SER10ac0d01 323mysys# camcontrol inquiry da2 -S 324ECNTX0LUN000000SER10ac0d01 325.Ed 326.Pp 327Now that you have used the Serial Number to compare two disk paths 328it is not entirely unreasonable to conclude that these are multiple 329paths to the same device. 330However, only the user who is familiar 331with their storage is qualified to make this judgement. 332.Pp 333You can then use the 334.Nm 335command to label and create a 336.Nm MULTIPATH 337GEOM provider named 338.Ar FRED . 339.Bd -literal -offset indent 340gmultipath label -v FRED /dev/da0 /dev/da2 341disklabel -Bw /dev/multipath/FRED auto 342newfs /dev/multipath/FREDa 343mount /dev/multipath/FREDa /mnt.... 344.Ed 345.Pp 346The resultant console output looks something like: 347.Bd -literal -offset indent 348GEOM_MULTIPATH: da0 added to FRED 349GEOM_MULTIPATH: da0 is now active path in FRED 350GEOM_MULTIPATH: da2 added to FRED 351.Ed 352.Pp 353To load the 354.Nm 355module at boot time, add this entry to 356.Pa /boot/loader.conf : 357.Bd -literal -offset ident 358geom_multipath_load="YES" 359.Ed 360.Sh SEE ALSO 361.Xr geom 4 , 362.Xr isp 4 , 363.Xr mpt 4 , 364.Xr loader.conf 5 , 365.Xr camcontrol 8 , 366.Xr geom 8 , 367.Xr mount 8 , 368.Xr newfs 8 , 369.Xr sysctl 8 370.Sh HISTORY 371The 372.Nm 373utility first appeared in 374.Fx 7.0 375.Sh AUTHORS 376.An Matthew Jacob Aq Mt mjacob@FreeBSD.org 377.An Alexander Motin Aq Mt mav@FreeBSD.org 378