1======= 2dm-raid 3======= 4 5The device-mapper RAID (dm-raid) target provides a bridge from DM to MD. 6It allows the MD RAID drivers to be accessed using a device-mapper 7interface. 8 9 10Mapping Table Interface 11----------------------- 12The target is named "raid" and it accepts the following parameters:: 13 14 <raid_type> <#raid_params> <raid_params> \ 15 <#raid_devs> <metadata_dev0> <dev0> [.. <metadata_devN> <devN>] 16 17<raid_type>: 18 19 ============= =============================================================== 20 raid0 RAID0 striping (no resilience) 21 raid1 RAID1 mirroring 22 raid4 RAID4 with dedicated last parity disk 23 raid5_n RAID5 with dedicated last parity disk supporting takeover 24 Same as raid4 25 26 - Transitory layout 27 raid5_la RAID5 left asymmetric 28 29 - rotating parity 0 with data continuation 30 raid5_ra RAID5 right asymmetric 31 32 - rotating parity N with data continuation 33 raid5_ls RAID5 left symmetric 34 35 - rotating parity 0 with data restart 36 raid5_rs RAID5 right symmetric 37 38 - rotating parity N with data restart 39 raid6_zr RAID6 zero restart 40 41 - rotating parity zero (left-to-right) with data restart 42 raid6_nr RAID6 N restart 43 44 - rotating parity N (right-to-left) with data restart 45 raid6_nc RAID6 N continue 46 47 - rotating parity N (right-to-left) with data continuation 48 raid6_n_6 RAID6 with dedicate parity disks 49 50 - parity and Q-syndrome on the last 2 disks; 51 layout for takeover from/to raid4/raid5_n 52 raid6_la_6 Same as "raid_la" plus dedicated last Q-syndrome disk 53 54 - layout for takeover from raid5_la from/to raid6 55 raid6_ra_6 Same as "raid5_ra" dedicated last Q-syndrome disk 56 57 - layout for takeover from raid5_ra from/to raid6 58 raid6_ls_6 Same as "raid5_ls" dedicated last Q-syndrome disk 59 60 - layout for takeover from raid5_ls from/to raid6 61 raid6_rs_6 Same as "raid5_rs" dedicated last Q-syndrome disk 62 63 - layout for takeover from raid5_rs from/to raid6 64 raid10 Various RAID10 inspired algorithms chosen by additional params 65 (see raid10_format and raid10_copies below) 66 67 - RAID10: Striped Mirrors (aka 'Striping on top of mirrors') 68 - RAID1E: Integrated Adjacent Stripe Mirroring 69 - RAID1E: Integrated Offset Stripe Mirroring 70 - and other similar RAID10 variants 71 ============= =============================================================== 72 73 Reference: Chapter 4 of 74 https://www.snia.org/sites/default/files/SNIA_DDF_Technical_Position_v2.0.pdf 75 76<#raid_params>: The number of parameters that follow. 77 78<raid_params> consists of 79 80 Mandatory parameters: 81 <chunk_size>: 82 Chunk size in sectors. This parameter is often known as 83 "stripe size". It is the only mandatory parameter and 84 is placed first. 85 86 followed by optional parameters (in any order): 87 [sync|nosync] 88 Force or prevent RAID initialization. 89 90 [rebuild <idx>] 91 Rebuild drive number 'idx' (first drive is 0). 92 93 [daemon_sleep <ms>] 94 Interval between runs of the bitmap daemon that 95 clear bits. A longer interval means less bitmap I/O but 96 resyncing after a failure is likely to take longer. 97 98 [min_recovery_rate <kB/sec/disk>] 99 Throttle RAID initialization 100 [max_recovery_rate <kB/sec/disk>] 101 Throttle RAID initialization 102 [write_mostly <idx>] 103 Mark drive index 'idx' write-mostly. 104 [max_write_behind <sectors>] 105 See '--write-behind=' (man mdadm) 106 [stripe_cache <sectors>] 107 Stripe cache size (RAID 4/5/6 only) 108 [region_size <sectors>] 109 The region_size multiplied by the number of regions is the 110 logical size of the array. The bitmap records the device 111 synchronisation state for each region. 112 113 [raid10_copies <# copies>], [raid10_format <near|far|offset>] 114 These two options are used to alter the default layout of 115 a RAID10 configuration. The number of copies is can be 116 specified, but the default is 2. There are also three 117 variations to how the copies are laid down - the default 118 is "near". Near copies are what most people think of with 119 respect to mirroring. If these options are left unspecified, 120 or 'raid10_copies 2' and/or 'raid10_format near' are given, 121 then the layouts for 2, 3 and 4 devices are: 122 123 ======== ========== ============== 124 2 drives 3 drives 4 drives 125 ======== ========== ============== 126 A1 A1 A1 A1 A2 A1 A1 A2 A2 127 A2 A2 A2 A3 A3 A3 A3 A4 A4 128 A3 A3 A4 A4 A5 A5 A5 A6 A6 129 A4 A4 A5 A6 A6 A7 A7 A8 A8 130 .. .. .. .. .. .. .. .. .. 131 ======== ========== ============== 132 133 The 2-device layout is equivalent 2-way RAID1. The 4-device 134 layout is what a traditional RAID10 would look like. The 135 3-device layout is what might be called a 'RAID1E - Integrated 136 Adjacent Stripe Mirroring'. 137 138 If 'raid10_copies 2' and 'raid10_format far', then the layouts 139 for 2, 3 and 4 devices are: 140 141 ======== ============ =================== 142 2 drives 3 drives 4 drives 143 ======== ============ =================== 144 A1 A2 A1 A2 A3 A1 A2 A3 A4 145 A3 A4 A4 A5 A6 A5 A6 A7 A8 146 A5 A6 A7 A8 A9 A9 A10 A11 A12 147 .. .. .. .. .. .. .. .. .. 148 A2 A1 A3 A1 A2 A2 A1 A4 A3 149 A4 A3 A6 A4 A5 A6 A5 A8 A7 150 A6 A5 A9 A7 A8 A10 A9 A12 A11 151 .. .. .. .. .. .. .. .. .. 152 ======== ============ =================== 153 154 If 'raid10_copies 2' and 'raid10_format offset', then the 155 layouts for 2, 3 and 4 devices are: 156 157 ======== ========== ================ 158 2 drives 3 drives 4 drives 159 ======== ========== ================ 160 A1 A2 A1 A2 A3 A1 A2 A3 A4 161 A2 A1 A3 A1 A2 A2 A1 A4 A3 162 A3 A4 A4 A5 A6 A5 A6 A7 A8 163 A4 A3 A6 A4 A5 A6 A5 A8 A7 164 A5 A6 A7 A8 A9 A9 A10 A11 A12 165 A6 A5 A9 A7 A8 A10 A9 A12 A11 166 .. .. .. .. .. .. .. .. .. 167 ======== ========== ================ 168 169 Here we see layouts closely akin to 'RAID1E - Integrated 170 Offset Stripe Mirroring'. 171 172 [delta_disks <N>] 173 The delta_disks option value (-251 < N < +251) triggers 174 device removal (negative value) or device addition (positive 175 value) to any reshape supporting raid levels 4/5/6 and 10. 176 RAID levels 4/5/6 allow for addition of devices (metadata 177 and data device tuple), raid10_near and raid10_offset only 178 allow for device addition. raid10_far does not support any 179 reshaping at all. 180 A minimum of devices have to be kept to enforce resilience, 181 which is 3 devices for raid4/5 and 4 devices for raid6. 182 183 [data_offset <sectors>] 184 This option value defines the offset into each data device 185 where the data starts. This is used to provide out-of-place 186 reshaping space to avoid writing over data while 187 changing the layout of stripes, hence an interruption/crash 188 may happen at any time without the risk of losing data. 189 E.g. when adding devices to an existing raid set during 190 forward reshaping, the out-of-place space will be allocated 191 at the beginning of each raid device. The kernel raid4/5/6/10 192 MD personalities supporting such device addition will read the data from 193 the existing first stripes (those with smaller number of stripes) 194 starting at data_offset to fill up a new stripe with the larger 195 number of stripes, calculate the redundancy blocks (CRC/Q-syndrome) 196 and write that new stripe to offset 0. Same will be applied to all 197 N-1 other new stripes. This out-of-place scheme is used to change 198 the RAID type (i.e. the allocation algorithm) as well, e.g. 199 changing from raid5_ls to raid5_n. 200 201 [journal_dev <dev>] 202 This option adds a journal device to raid4/5/6 raid sets and 203 uses it to close the 'write hole' caused by the non-atomic updates 204 to the component devices which can cause data loss during recovery. 205 The journal device is used as writethrough thus causing writes to 206 be throttled versus non-journaled raid4/5/6 sets. 207 Takeover/reshape is not possible with a raid4/5/6 journal device; 208 it has to be deconfigured before requesting these. 209 210 [journal_mode <mode>] 211 This option sets the caching mode on journaled raid4/5/6 raid sets 212 (see 'journal_dev <dev>' above) to 'writethrough' or 'writeback'. 213 If 'writeback' is selected the journal device has to be resilient 214 and must not suffer from the 'write hole' problem itself (e.g. use 215 raid1 or raid10) to avoid a single point of failure. 216 217<#raid_devs>: The number of devices composing the array. 218 Each device consists of two entries. The first is the device 219 containing the metadata (if any); the second is the one containing the 220 data. A Maximum of 64 metadata/data device entries are supported 221 up to target version 1.8.0. 222 1.9.0 supports up to 253 which is enforced by the used MD kernel runtime. 223 224 If a drive has failed or is missing at creation time, a '-' can be 225 given for both the metadata and data drives for a given position. 226 227 228Example Tables 229-------------- 230 231:: 232 233 # RAID4 - 4 data drives, 1 parity (no metadata devices) 234 # No metadata devices specified to hold superblock/bitmap info 235 # Chunk size of 1MiB 236 # (Lines separated for easy reading) 237 238 0 1960893648 raid \ 239 raid4 1 2048 \ 240 5 - 8:17 - 8:33 - 8:49 - 8:65 - 8:81 241 242 # RAID4 - 4 data drives, 1 parity (with metadata devices) 243 # Chunk size of 1MiB, force RAID initialization, 244 # min recovery rate at 20 kiB/sec/disk 245 246 0 1960893648 raid \ 247 raid4 4 2048 sync min_recovery_rate 20 \ 248 5 8:17 8:18 8:33 8:34 8:49 8:50 8:65 8:66 8:81 8:82 249 250 251Status Output 252------------- 253'dmsetup table' displays the table used to construct the mapping. 254The optional parameters are always printed in the order listed 255above with "sync" or "nosync" always output ahead of the other 256arguments, regardless of the order used when originally loading the table. 257Arguments that can be repeated are ordered by value. 258 259 260'dmsetup status' yields information on the state and health of the array. 261The output is as follows (normally a single line, but expanded here for 262clarity):: 263 264 1: <s> <l> raid \ 265 2: <raid_type> <#devices> <health_chars> \ 266 3: <sync_ratio> <sync_action> <mismatch_cnt> 267 268Line 1 is the standard output produced by device-mapper. 269 270Line 2 & 3 are produced by the raid target and are best explained by example:: 271 272 0 1960893648 raid raid4 5 AAAAA 2/490221568 init 0 273 274Here we can see the RAID type is raid4, there are 5 devices - all of 275which are 'A'live, and the array is 2/490221568 complete with its initial 276recovery. Here is a fuller description of the individual fields: 277 278 =============== ========================================================= 279 <raid_type> Same as the <raid_type> used to create the array. 280 <health_chars> One char for each device, indicating: 281 282 - 'A' = alive and in-sync 283 - 'a' = alive but not in-sync 284 - 'D' = dead/failed. 285 <sync_ratio> The ratio indicating how much of the array has undergone 286 the process described by 'sync_action'. If the 287 'sync_action' is "check" or "repair", then the process 288 of "resync" or "recover" can be considered complete. 289 <sync_action> One of the following possible states: 290 291 idle 292 - No synchronization action is being performed. 293 frozen 294 - The current action has been halted. 295 resync 296 - Array is undergoing its initial synchronization 297 or is resynchronizing after an unclean shutdown 298 (possibly aided by a bitmap). 299 recover 300 - A device in the array is being rebuilt or 301 replaced. 302 check 303 - A user-initiated full check of the array is 304 being performed. All blocks are read and 305 checked for consistency. The number of 306 discrepancies found are recorded in 307 <mismatch_cnt>. No changes are made to the 308 array by this action. 309 repair 310 - The same as "check", but discrepancies are 311 corrected. 312 reshape 313 - The array is undergoing a reshape. 314 <mismatch_cnt> The number of discrepancies found between mirror copies 315 in RAID1/10 or wrong parity values found in RAID4/5/6. 316 This value is valid only after a "check" of the array 317 is performed. A healthy array has a 'mismatch_cnt' of 0. 318 <data_offset> The current data offset to the start of the user data on 319 each component device of a raid set (see the respective 320 raid parameter to support out-of-place reshaping). 321 <journal_char> - 'A' - active write-through journal device. 322 - 'a' - active write-back journal device. 323 - 'D' - dead journal device. 324 - '-' - no journal device. 325 =============== ========================================================= 326 327 328Message Interface 329----------------- 330The dm-raid target will accept certain actions through the 'message' interface. 331('man dmsetup' for more information on the message interface.) These actions 332include: 333 334 ========= ================================================ 335 "idle" Halt the current sync action. 336 "frozen" Freeze the current sync action. 337 "resync" Initiate/continue a resync. 338 "recover" Initiate/continue a recover process. 339 "check" Initiate a check (i.e. a "scrub") of the array. 340 "repair" Initiate a repair of the array. 341 ========= ================================================ 342 343 344Discard Support 345--------------- 346The implementation of discard support among hardware vendors varies. 347When a block is discarded, some storage devices will return zeroes when 348the block is read. These devices set the 'discard_zeroes_data' 349attribute. Other devices will return random data. Confusingly, some 350devices that advertise 'discard_zeroes_data' will not reliably return 351zeroes when discarded blocks are read! Since RAID 4/5/6 uses blocks 352from a number of devices to calculate parity blocks and (for performance 353reasons) relies on 'discard_zeroes_data' being reliable, it is important 354that the devices be consistent. Blocks may be discarded in the middle 355of a RAID 4/5/6 stripe and if subsequent read results are not 356consistent, the parity blocks may be calculated differently at any time; 357making the parity blocks useless for redundancy. It is important to 358understand how your hardware behaves with discards if you are going to 359enable discards with RAID 4/5/6. 360 361Since the behavior of storage devices is unreliable in this respect, 362even when reporting 'discard_zeroes_data', by default RAID 4/5/6 363discard support is disabled -- this ensures data integrity at the 364expense of losing some performance. 365 366Storage devices that properly support 'discard_zeroes_data' are 367increasingly whitelisted in the kernel and can thus be trusted. 368 369For trusted devices, the following dm-raid module parameter can be set 370to safely enable discard support for RAID 4/5/6: 371 372 'devices_handle_discards_safely' 373 374 375Version History 376--------------- 377 378:: 379 380 1.0.0 Initial version. Support for RAID 4/5/6 381 1.1.0 Added support for RAID 1 382 1.2.0 Handle creation of arrays that contain failed devices. 383 1.3.0 Added support for RAID 10 384 1.3.1 Allow device replacement/rebuild for RAID 10 385 1.3.2 Fix/improve redundancy checking for RAID10 386 1.4.0 Non-functional change. Removes arg from mapping function. 387 1.4.1 RAID10 fix redundancy validation checks (commit 55ebbb5). 388 1.4.2 Add RAID10 "far" and "offset" algorithm support. 389 1.5.0 Add message interface to allow manipulation of the sync_action. 390 New status (STATUSTYPE_INFO) fields: sync_action and mismatch_cnt. 391 1.5.1 Add ability to restore transiently failed devices on resume. 392 1.5.2 'mismatch_cnt' is zero unless [last_]sync_action is "check". 393 1.6.0 Add discard support (and devices_handle_discard_safely module param). 394 1.7.0 Add support for MD RAID0 mappings. 395 1.8.0 Explicitly check for compatible flags in the superblock metadata 396 and reject to start the raid set if any are set by a newer 397 target version, thus avoiding data corruption on a raid set 398 with a reshape in progress. 399 1.9.0 Add support for RAID level takeover/reshape/region size 400 and set size reduction. 401 1.9.1 Fix activation of existing RAID 4/10 mapped devices 402 1.9.2 Don't emit '- -' on the status table line in case the constructor 403 fails reading a superblock. Correctly emit 'maj:min1 maj:min2' and 404 'D' on the status line. If '- -' is passed into the constructor, emit 405 '- -' on the table line and '-' as the status line health character. 406 1.10.0 Add support for raid4/5/6 journal device 407 1.10.1 Fix data corruption on reshape request 408 1.11.0 Fix table line argument order 409 (wrong raid10_copies/raid10_format sequence) 410 1.11.1 Add raid4/5/6 journal write-back support via journal_mode option 411 1.12.1 Fix for MD deadlock between mddev_suspend() and md_write_start() available 412 1.13.0 Fix dev_health status at end of "recover" (was 'a', now 'A') 413 1.13.1 Fix deadlock caused by early md_stop_writes(). Also fix size an 414 state races. 415 1.13.2 Fix raid redundancy validation and avoid keeping raid set frozen 416 1.14.0 Fix reshape race on small devices. Fix stripe adding reshape 417 deadlock/potential data corruption. Update superblock when 418 specific devices are requested via rebuild. Fix RAID leg 419 rebuild errors. 420 1.15.0 Fix size extensions not being synchronized in case of new MD bitmap 421 pages allocated; also fix those not occurring after previous reductions 422 1.15.1 Fix argument count and arguments for rebuild/write_mostly/journal_(dev|mode) 423 on the status line. 424