xref: /freebsd/lib/geom/multipath/gmultipath.8 (revision 6137b5f7b8c183ee8806d79b3f1d8e5e3ddb3df3)
1.\" Copyright (c) 2007 Matthew Jacob
2.\" All rights reserved.
3.\"
4.\" Redistribution and use in source and binary forms, with or without
5.\" modification, are permitted provided that the following conditions
6.\" are met:
7.\" 1. Redistributions of source code must retain the above copyright
8.\"    notice, this list of conditions and the following disclaimer.
9.\" 2. Redistributions in binary form must reproduce the above copyright
10.\"    notice, this list of conditions and the following disclaimer in the
11.\"    documentation and/or other materials provided with the distribution.
12.\"
13.\" THIS SOFTWARE IS PROVIDED BY THE AUTHORS AND CONTRIBUTORS ``AS IS'' AND
14.\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
15.\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
16.\" ARE DISCLAIMED.  IN NO EVENT SHALL THE AUTHORS OR CONTRIBUTORS BE LIABLE
17.\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
18.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
19.\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
20.\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
21.\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
22.\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
23.\" SUCH DAMAGE.
24.\"
25.Dd March 17, 2022
26.Dt GMULTIPATH 8
27.Os
28.Sh NAME
29.Nm gmultipath
30.Nd "disk multipath control utility"
31.Sh SYNOPSIS
32.Nm
33.Cm create
34.Op Fl ARv
35.Ar name
36.Ar prov ...
37.Nm
38.Cm label
39.Op Fl ARv
40.Ar name
41.Ar prov ...
42.Nm
43.Cm configure
44.Op Fl APRv
45.Ar name
46.Nm
47.Cm add
48.Op Fl v
49.Ar name prov
50.Nm
51.Cm remove
52.Op Fl v
53.Ar name prov
54.Nm
55.Cm fail
56.Op Fl v
57.Ar name prov
58.Nm
59.Cm restore
60.Op Fl v
61.Ar name prov
62.Nm
63.Cm rotate
64.Op Fl v
65.Ar name
66.Nm
67.Cm prefer
68.Op Fl v
69.Ar name
70.Ar prov
71.Nm
72.Cm getactive
73.Op Fl v
74.Ar name
75.Nm
76.Cm destroy
77.Op Fl v
78.Ar name
79.Nm
80.Cm stop
81.Op Fl v
82.Ar name
83.Nm
84.Cm clear
85.Op Fl v
86.Ar prov ...
87.Nm
88.Cm list
89.Nm
90.Cm status
91.Nm
92.Cm load
93.Nm
94.Cm unload
95.Sh DESCRIPTION
96The
97.Nm
98utility is used for device multipath configuration.
99.Pp
100The multipath device can be configured using two different methods:
101.Dq manual
102or
103.Dq automatic .
104When using the
105.Dq manual
106method, no metadata are stored on the devices, so the multipath
107device has to be configured by hand every time it is needed.
108Additional device paths also will not be detected automatically.
109The
110.Dq automatic
111method uses on-disk metadata to detect device and all its paths.
112Metadata use the last sector of the underlying disk device and
113include device name and UUID.
114The UUID guarantees uniqueness in a shared storage environment
115but is in general too cumbersome to use.
116The name is what is exported via the device interface.
117.Pp
118The first argument to
119.Nm
120indicates an action to be performed:
121.Bl -tag -width ".Cm destroy"
122.It Cm create
123Create multipath device with
124.Dq manual
125method without writing any on-disk metadata.
126It is up to administrator, how to properly identify device paths.
127Kernel will only check that all given providers have same media and
128sector sizes.
129.Pp
130.Fl A
131option enables Active/Active mode,
132.Fl R
133option enables Active/Read mode, otherwise Active/Passive mode is used
134by default.
135.It Cm label
136Create multipath device with
137.Dq automatic
138method.
139Label the first given provider with on-disk metadata using the specified
140.Ar name .
141The rest of given providers will be retasted to detect these metadata.
142It reliably protects against specifying unrelated providers.
143Providers with no matching metadata detected will not be added to the device.
144.Pp
145.Fl A
146option enables Active/Active mode,
147.Fl R
148option enables Active/Read mode, otherwise Active/Passive mode is used
149by default.
150.It Cm configure
151Configure the given multipath device.
152.Pp
153.Fl A
154option enables Active/Active mode,
155.Fl P
156option enables Active/Passive mode,
157.Fl R
158option enables Active/Read mode.
159.It Cm add
160Add the given provider as a path to the given multipath device.
161Should normally be used only for devices created with
162.Dq manual
163method, unless you know what you are doing (you are sure that it is another
164device path, but tasting its metadata in regular
165.Dq automatic
166way is not possible).
167.It Cm remove
168Remove the given provider as a path from the given multipath device.
169If the last path removed, the multipath device will be destroyed.
170.It Cm fail
171Mark specified provider as a path of the specified multipath device as failed.
172If there are other paths present, new requests will be forwarded there.
173.It Cm restore
174Mark specified provider as a path of the specified multipath device as
175operational, allowing it to handle requests.
176.It Cm rotate
177Change the active provider/path to the next available provider in Active/Passive mode.
178.It Cm prefer
179Change the active provider/path to the specified provider in Active/Passive mode.
180.It Cm getactive
181Get the currently active provider(s)/path(s).
182.It Cm destroy
183Destroy the given multipath device clearing metadata.
184.It Cm stop
185Stop the given multipath device without clearing metadata.
186.It Cm clear
187Clear metadata on the given provider.
188.It Cm list
189See
190.Xr geom 8 .
191.It Cm status
192See
193.Xr geom 8 .
194.It Cm load
195See
196.Xr geom 8 .
197.It Cm unload
198See
199.Xr geom 8 .
200.El
201.Sh SYSCTL VARIABLES
202The following
203.Xr sysctl 8
204variable can be used to control the behavior of the
205.Nm MULTIPATH
206GEOM class.
207.Bl -tag -width indent
208.It Va kern.geom.multipath.debug : No 0
209Debug level of the
210.Nm MULTIPATH
211GEOM class.
212This can be set to 0 (default) or 1 to disable or enable various
213forms of chattiness.
214.It Va kern.geom.multipath.exclusive : No 1
215Open underlying providers exclusively, preventing individual paths access.
216.El
217.Sh EXIT STATUS
218Exit status is 0 on success, and 1 if the command fails.
219.Sh MULTIPATH ARCHITECTURE
220This is a multiple path architecture with no device knowledge or
221presumptions other than size matching built in.
222Therefore the user must exercise some care
223in selecting providers that do indeed represent multiple paths to the
224same underlying disk device.
225The reason for this is that there are several
226criteria across multiple underlying transport types that can
227.Ar indicate
228identity, but in all respects such identity can rarely be considered
229.Ar definitive .
230.Pp
231For example, if you use the World Wide Port Name of a Fibre Channel
232disk object you might believe that two disks that have the same WWPN
233on different paths (or even disjoint fabrics) might be considered
234the same disk.
235Nearly always this would be a safe assumption, until
236you realize that a WWPN, like an Ethernet MAC address, is a soft
237programmable entity, and that a misconfigured Director Class switch
238could lead you to believe incorrectly that you have found multiple
239paths to the same device.
240This is an extreme and theoretical case, but
241it is possible enough to indicate that the policy for deciding which
242of multiple pathnames refer to the same device should be left to the
243system operator who will use tools and knowledge of their own storage
244subsystem to make the correct configuration selection.
245.Pp
246There are Active/Passive, Active/Read and Active/Active operation modes
247supported.
248In Active/Passive mode only one path has I/O moving on it
249at any point in time.
250This I/O continues until an I/O is returned with
251a generic I/O error or a "Nonexistent Device" error.
252When this occurs, that path is marked FAIL, the next path
253in a list is selected as active and the failed I/O reissued.
254In Active/Active mode all paths not marked FAIL may handle I/O at the same time.
255Requests are distributed between paths to equalize load.
256For capable devices it allows the utilisation of the bandwidth available on all paths.
257In Active/Read mode all paths not marked FAIL may handle reads at the same time,
258but unlike in Active/Active mode only one path handles write requests at any
259point in time; closely following the original write request order if the layer
260above needs it for data consistency (not waiting for requisite write completion
261before sending dependent write).
262.Pp
263When new devices are added to the system the
264.Nm MULTIPATH
265GEOM class is given an opportunity to taste these new devices.
266If a new
267device has a
268.Nm MULTIPATH
269on-disk metadata label, the device is either used to create a new
270.Nm MULTIPATH
271GEOM, or added to the list of paths for an existing
272.Nm MULTIPATH
273GEOM.
274.Pp
275It is this mechanism that works reasonably with
276.Xr isp 4
277and
278.Xr mpt 4
279based Fibre Channel disk devices.
280For these devices, when a device disappears
281(due to e.g., a cable pull or power failure to a switch), the device is
282proactively marked as gone and I/O to it failed.
283This causes the
284.Nm MULTIPATH
285failure event just described.
286.Pp
287When Fibre Channel events inform either
288.Xr isp 4
289or
290.Xr mpt 4
291host bus adapters that new devices may have arrived (e.g., the arrival
292of an RSCN event from the Fabric Domain Controller), they can cause
293a rescan to occur and cause the attachment and configuration of any
294(now) new devices to occur, causing the taste event described above.
295.Pp
296This means that this multipath architecture is not a one-shot path
297failover, but can be considered to be steady state as long as failed
298paths are repaired (automatically or otherwise).
299.Pp
300Automatic rescanning is not a requirement.
301Nor is Fibre Channel.
302The
303same failover mechanisms work equally well for traditional "Parallel"
304SCSI but may require manual intervention with
305.Xr camcontrol 8
306to cause the reattachment of repaired device links.
307.Sh EXAMPLES
308The following example shows how to use
309.Xr camcontrol 8
310to find possible multiple path devices and to create a
311.Nm MULTIPATH
312GEOM class for them.
313.Bd -literal -offset indent
314mysys# camcontrol devlist
315<ECNCTX @WESTVILLE >   at scbus0 target 0 lun 0 (da0,pass0)
316<ECNCTX @WESTVILLE >   at scbus0 target 0 lun 1 (da1,pass1)
317<ECNCTX @WESTVILLE >   at scbus1 target 0 lun 0 (da2,pass2)
318<ECNCTX @WESTVILLE >   at scbus1 target 0 lun 1 (da3,pass3)
319mysys# camcontrol inquiry da0 -S
320ECNTX0LUN000000SER10ac0d01
321mysys# camcontrol inquiry da2 -S
322ECNTX0LUN000000SER10ac0d01
323.Ed
324.Pp
325Now that you have used the Serial Number to compare two disk paths
326it is not entirely unreasonable to conclude that these are multiple
327paths to the same device.
328However, only the user who is familiar
329with their storage is qualified to make this judgement.
330.Pp
331You can then use the
332.Nm
333command to label and create a
334.Nm MULTIPATH
335GEOM provider named
336.Ar FRED .
337.Bd -literal -offset indent
338gmultipath label -v FRED /dev/da0 /dev/da2
339disklabel -Bw /dev/multipath/FRED auto
340newfs /dev/multipath/FREDa
341mount /dev/multipath/FREDa /mnt....
342.Ed
343.Pp
344The resultant console output looks something like:
345.Bd -literal -offset indent
346GEOM_MULTIPATH: da0 added to FRED
347GEOM_MULTIPATH: da0 is now active path in FRED
348GEOM_MULTIPATH: da2 added to FRED
349.Ed
350.Pp
351To load the
352.Nm
353module at boot time, add this entry to
354.Pa /boot/loader.conf :
355.Bd -literal -offset ident
356geom_multipath_load="YES"
357.Ed
358.Sh SEE ALSO
359.Xr geom 4 ,
360.Xr isp 4 ,
361.Xr mpt 4 ,
362.Xr loader.conf 5 ,
363.Xr camcontrol 8 ,
364.Xr geom 8 ,
365.Xr mount 8 ,
366.Xr newfs 8 ,
367.Xr sysctl 8
368.Sh HISTORY
369The
370.Nm
371utility first appeared in
372.Fx 7.0
373.Sh AUTHORS
374.An Matthew Jacob Aq Mt mjacob@FreeBSD.org
375.An Alexander Motin Aq Mt mav@FreeBSD.org
376