1=============================== 2LIBNVDIMM: Non-Volatile Devices 3=============================== 4 5libnvdimm - kernel / libndctl - userspace helper library 6 7linux-nvdimm@lists.01.org 8 9Version 13 10 11.. contents: 12 13 Glossary 14 Overview 15 Supporting Documents 16 Git Trees 17 LIBNVDIMM PMEM and BLK 18 Why BLK? 19 PMEM vs BLK 20 BLK-REGIONs, PMEM-REGIONs, Atomic Sectors, and DAX 21 Example NVDIMM Platform 22 LIBNVDIMM Kernel Device Model and LIBNDCTL Userspace API 23 LIBNDCTL: Context 24 libndctl: instantiate a new library context example 25 LIBNVDIMM/LIBNDCTL: Bus 26 libnvdimm: control class device in /sys/class 27 libnvdimm: bus 28 libndctl: bus enumeration example 29 LIBNVDIMM/LIBNDCTL: DIMM (NMEM) 30 libnvdimm: DIMM (NMEM) 31 libndctl: DIMM enumeration example 32 LIBNVDIMM/LIBNDCTL: Region 33 libnvdimm: region 34 libndctl: region enumeration example 35 Why Not Encode the Region Type into the Region Name? 36 How Do I Determine the Major Type of a Region? 37 LIBNVDIMM/LIBNDCTL: Namespace 38 libnvdimm: namespace 39 libndctl: namespace enumeration example 40 libndctl: namespace creation example 41 Why the Term "namespace"? 42 LIBNVDIMM/LIBNDCTL: Block Translation Table "btt" 43 libnvdimm: btt layout 44 libndctl: btt creation example 45 Summary LIBNDCTL Diagram 46 47 48Glossary 49======== 50 51PMEM: 52 A system-physical-address range where writes are persistent. A 53 block device composed of PMEM is capable of DAX. A PMEM address range 54 may span an interleave of several DIMMs. 55 56BLK: 57 A set of one or more programmable memory mapped apertures provided 58 by a DIMM to access its media. This indirection precludes the 59 performance benefit of interleaving, but enables DIMM-bounded failure 60 modes. 61 62DPA: 63 DIMM Physical Address, is a DIMM-relative offset. With one DIMM in 64 the system there would be a 1:1 system-physical-address:DPA association. 65 Once more DIMMs are added a memory controller interleave must be 66 decoded to determine the DPA associated with a given 67 system-physical-address. BLK capacity always has a 1:1 relationship 68 with a single-DIMM's DPA range. 69 70DAX: 71 File system extensions to bypass the page cache and block layer to 72 mmap persistent memory, from a PMEM block device, directly into a 73 process address space. 74 75DSM: 76 Device Specific Method: ACPI method to control specific 77 device - in this case the firmware. 78 79DCR: 80 NVDIMM Control Region Structure defined in ACPI 6 Section 5.2.25.5. 81 It defines a vendor-id, device-id, and interface format for a given DIMM. 82 83BTT: 84 Block Translation Table: Persistent memory is byte addressable. 85 Existing software may have an expectation that the power-fail-atomicity 86 of writes is at least one sector, 512 bytes. The BTT is an indirection 87 table with atomic update semantics to front a PMEM/BLK block device 88 driver and present arbitrary atomic sector sizes. 89 90LABEL: 91 Metadata stored on a DIMM device that partitions and identifies 92 (persistently names) storage between PMEM and BLK. It also partitions 93 BLK storage to host BTTs with different parameters per BLK-partition. 94 Note that traditional partition tables, GPT/MBR, are layered on top of a 95 BLK or PMEM device. 96 97 98Overview 99======== 100 101The LIBNVDIMM subsystem provides support for three types of NVDIMMs, namely, 102PMEM, BLK, and NVDIMM devices that can simultaneously support both PMEM 103and BLK mode access. These three modes of operation are described by 104the "NVDIMM Firmware Interface Table" (NFIT) in ACPI 6. While the LIBNVDIMM 105implementation is generic and supports pre-NFIT platforms, it was guided 106by the superset of capabilities need to support this ACPI 6 definition 107for NVDIMM resources. The bulk of the kernel implementation is in place 108to handle the case where DPA accessible via PMEM is aliased with DPA 109accessible via BLK. When that occurs a LABEL is needed to reserve DPA 110for exclusive access via one mode a time. 111 112Supporting Documents 113-------------------- 114 115ACPI 6: 116 https://www.uefi.org/sites/default/files/resources/ACPI_6.0.pdf 117NVDIMM Namespace: 118 https://pmem.io/documents/NVDIMM_Namespace_Spec.pdf 119DSM Interface Example: 120 https://pmem.io/documents/NVDIMM_DSM_Interface_Example.pdf 121Driver Writer's Guide: 122 https://pmem.io/documents/NVDIMM_Driver_Writers_Guide.pdf 123 124Git Trees 125--------- 126 127LIBNVDIMM: 128 https://git.kernel.org/cgit/linux/kernel/git/djbw/nvdimm.git 129LIBNDCTL: 130 https://github.com/pmem/ndctl.git 131PMEM: 132 https://github.com/01org/prd 133 134 135LIBNVDIMM PMEM and BLK 136====================== 137 138Prior to the arrival of the NFIT, non-volatile memory was described to a 139system in various ad-hoc ways. Usually only the bare minimum was 140provided, namely, a single system-physical-address range where writes 141are expected to be durable after a system power loss. Now, the NFIT 142specification standardizes not only the description of PMEM, but also 143BLK and platform message-passing entry points for control and 144configuration. 145 146For each NVDIMM access method (PMEM, BLK), LIBNVDIMM provides a block 147device driver: 148 149 1. PMEM (nd_pmem.ko): Drives a system-physical-address range. This 150 range is contiguous in system memory and may be interleaved (hardware 151 memory controller striped) across multiple DIMMs. When interleaved the 152 platform may optionally provide details of which DIMMs are participating 153 in the interleave. 154 155 Note that while LIBNVDIMM describes system-physical-address ranges that may 156 alias with BLK access as ND_NAMESPACE_PMEM ranges and those without 157 alias as ND_NAMESPACE_IO ranges, to the nd_pmem driver there is no 158 distinction. The different device-types are an implementation detail 159 that userspace can exploit to implement policies like "only interface 160 with address ranges from certain DIMMs". It is worth noting that when 161 aliasing is present and a DIMM lacks a label, then no block device can 162 be created by default as userspace needs to do at least one allocation 163 of DPA to the PMEM range. In contrast ND_NAMESPACE_IO ranges, once 164 registered, can be immediately attached to nd_pmem. 165 166 2. BLK (nd_blk.ko): This driver performs I/O using a set of platform 167 defined apertures. A set of apertures will access just one DIMM. 168 Multiple windows (apertures) allow multiple concurrent accesses, much like 169 tagged-command-queuing, and would likely be used by different threads or 170 different CPUs. 171 172 The NFIT specification defines a standard format for a BLK-aperture, but 173 the spec also allows for vendor specific layouts, and non-NFIT BLK 174 implementations may have other designs for BLK I/O. For this reason 175 "nd_blk" calls back into platform-specific code to perform the I/O. 176 177 One such implementation is defined in the "Driver Writer's Guide" and "DSM 178 Interface Example". 179 180 181Why BLK? 182======== 183 184While PMEM provides direct byte-addressable CPU-load/store access to 185NVDIMM storage, it does not provide the best system RAS (recovery, 186availability, and serviceability) model. An access to a corrupted 187system-physical-address address causes a CPU exception while an access 188to a corrupted address through an BLK-aperture causes that block window 189to raise an error status in a register. The latter is more aligned with 190the standard error model that host-bus-adapter attached disks present. 191 192Also, if an administrator ever wants to replace a memory it is easier to 193service a system at DIMM module boundaries. Compare this to PMEM where 194data could be interleaved in an opaque hardware specific manner across 195several DIMMs. 196 197PMEM vs BLK 198----------- 199 200BLK-apertures solve these RAS problems, but their presence is also the 201major contributing factor to the complexity of the ND subsystem. They 202complicate the implementation because PMEM and BLK alias in DPA space. 203Any given DIMM's DPA-range may contribute to one or more 204system-physical-address sets of interleaved DIMMs, *and* may also be 205accessed in its entirety through its BLK-aperture. Accessing a DPA 206through a system-physical-address while simultaneously accessing the 207same DPA through a BLK-aperture has undefined results. For this reason, 208DIMMs with this dual interface configuration include a DSM function to 209store/retrieve a LABEL. The LABEL effectively partitions the DPA-space 210into exclusive system-physical-address and BLK-aperture accessible 211regions. For simplicity a DIMM is allowed a PMEM "region" per each 212interleave set in which it is a member. The remaining DPA space can be 213carved into an arbitrary number of BLK devices with discontiguous 214extents. 215 216BLK-REGIONs, PMEM-REGIONs, Atomic Sectors, and DAX 217^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 218 219One of the few 220reasons to allow multiple BLK namespaces per REGION is so that each 221BLK-namespace can be configured with a BTT with unique atomic sector 222sizes. While a PMEM device can host a BTT the LABEL specification does 223not provide for a sector size to be specified for a PMEM namespace. 224 225This is due to the expectation that the primary usage model for PMEM is 226via DAX, and the BTT is incompatible with DAX. However, for the cases 227where an application or filesystem still needs atomic sector update 228guarantees it can register a BTT on a PMEM device or partition. See 229LIBNVDIMM/NDCTL: Block Translation Table "btt" 230 231 232Example NVDIMM Platform 233======================= 234 235For the remainder of this document the following diagram will be 236referenced for any example sysfs layouts:: 237 238 239 (a) (b) DIMM BLK-REGION 240 +-------------------+--------+--------+--------+ 241 +------+ | pm0.0 | blk2.0 | pm1.0 | blk2.1 | 0 region2 242 | imc0 +--+- - - region0- - - +--------+ +--------+ 243 +--+---+ | pm0.0 | blk3.0 | pm1.0 | blk3.1 | 1 region3 244 | +-------------------+--------v v--------+ 245 +--+---+ | | 246 | cpu0 | region1 247 +--+---+ | | 248 | +----------------------------^ ^--------+ 249 +--+---+ | blk4.0 | pm1.0 | blk4.0 | 2 region4 250 | imc1 +--+----------------------------| +--------+ 251 +------+ | blk5.0 | pm1.0 | blk5.0 | 3 region5 252 +----------------------------+--------+--------+ 253 254In this platform we have four DIMMs and two memory controllers in one 255socket. Each unique interface (BLK or PMEM) to DPA space is identified 256by a region device with a dynamically assigned id (REGION0 - REGION5). 257 258 1. The first portion of DIMM0 and DIMM1 are interleaved as REGION0. A 259 single PMEM namespace is created in the REGION0-SPA-range that spans most 260 of DIMM0 and DIMM1 with a user-specified name of "pm0.0". Some of that 261 interleaved system-physical-address range is reclaimed as BLK-aperture 262 accessed space starting at DPA-offset (a) into each DIMM. In that 263 reclaimed space we create two BLK-aperture "namespaces" from REGION2 and 264 REGION3 where "blk2.0" and "blk3.0" are just human readable names that 265 could be set to any user-desired name in the LABEL. 266 267 2. In the last portion of DIMM0 and DIMM1 we have an interleaved 268 system-physical-address range, REGION1, that spans those two DIMMs as 269 well as DIMM2 and DIMM3. Some of REGION1 is allocated to a PMEM namespace 270 named "pm1.0", the rest is reclaimed in 4 BLK-aperture namespaces (for 271 each DIMM in the interleave set), "blk2.1", "blk3.1", "blk4.0", and 272 "blk5.0". 273 274 3. The portion of DIMM2 and DIMM3 that do not participate in the REGION1 275 interleaved system-physical-address range (i.e. the DPA address past 276 offset (b) are also included in the "blk4.0" and "blk5.0" namespaces. 277 Note, that this example shows that BLK-aperture namespaces don't need to 278 be contiguous in DPA-space. 279 280 This bus is provided by the kernel under the device 281 /sys/devices/platform/nfit_test.0 when the nfit_test.ko module from 282 tools/testing/nvdimm is loaded. This not only test LIBNVDIMM but the 283 acpi_nfit.ko driver as well. 284 285 286LIBNVDIMM Kernel Device Model and LIBNDCTL Userspace API 287======================================================== 288 289What follows is a description of the LIBNVDIMM sysfs layout and a 290corresponding object hierarchy diagram as viewed through the LIBNDCTL 291API. The example sysfs paths and diagrams are relative to the Example 292NVDIMM Platform which is also the LIBNVDIMM bus used in the LIBNDCTL unit 293test. 294 295LIBNDCTL: Context 296----------------- 297 298Every API call in the LIBNDCTL library requires a context that holds the 299logging parameters and other library instance state. The library is 300based on the libabc template: 301 302 https://git.kernel.org/cgit/linux/kernel/git/kay/libabc.git 303 304LIBNDCTL: instantiate a new library context example 305^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 306 307:: 308 309 struct ndctl_ctx *ctx; 310 311 if (ndctl_new(&ctx) == 0) 312 return ctx; 313 else 314 return NULL; 315 316LIBNVDIMM/LIBNDCTL: Bus 317----------------------- 318 319A bus has a 1:1 relationship with an NFIT. The current expectation for 320ACPI based systems is that there is only ever one platform-global NFIT. 321That said, it is trivial to register multiple NFITs, the specification 322does not preclude it. The infrastructure supports multiple busses and 323we use this capability to test multiple NFIT configurations in the unit 324test. 325 326LIBNVDIMM: control class device in /sys/class 327--------------------------------------------- 328 329This character device accepts DSM messages to be passed to DIMM 330identified by its NFIT handle:: 331 332 /sys/class/nd/ndctl0 333 |-- dev 334 |-- device -> ../../../ndbus0 335 |-- subsystem -> ../../../../../../../class/nd 336 337 338 339LIBNVDIMM: bus 340-------------- 341 342:: 343 344 struct nvdimm_bus *nvdimm_bus_register(struct device *parent, 345 struct nvdimm_bus_descriptor *nfit_desc); 346 347:: 348 349 /sys/devices/platform/nfit_test.0/ndbus0 350 |-- commands 351 |-- nd 352 |-- nfit 353 |-- nmem0 354 |-- nmem1 355 |-- nmem2 356 |-- nmem3 357 |-- power 358 |-- provider 359 |-- region0 360 |-- region1 361 |-- region2 362 |-- region3 363 |-- region4 364 |-- region5 365 |-- uevent 366 `-- wait_probe 367 368LIBNDCTL: bus enumeration example 369^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 370 371Find the bus handle that describes the bus from Example NVDIMM Platform:: 372 373 static struct ndctl_bus *get_bus_by_provider(struct ndctl_ctx *ctx, 374 const char *provider) 375 { 376 struct ndctl_bus *bus; 377 378 ndctl_bus_foreach(ctx, bus) 379 if (strcmp(provider, ndctl_bus_get_provider(bus)) == 0) 380 return bus; 381 382 return NULL; 383 } 384 385 bus = get_bus_by_provider(ctx, "nfit_test.0"); 386 387 388LIBNVDIMM/LIBNDCTL: DIMM (NMEM) 389------------------------------- 390 391The DIMM device provides a character device for sending commands to 392hardware, and it is a container for LABELs. If the DIMM is defined by 393NFIT then an optional 'nfit' attribute sub-directory is available to add 394NFIT-specifics. 395 396Note that the kernel device name for "DIMMs" is "nmemX". The NFIT 397describes these devices via "Memory Device to System Physical Address 398Range Mapping Structure", and there is no requirement that they actually 399be physical DIMMs, so we use a more generic name. 400 401LIBNVDIMM: DIMM (NMEM) 402^^^^^^^^^^^^^^^^^^^^^^ 403 404:: 405 406 struct nvdimm *nvdimm_create(struct nvdimm_bus *nvdimm_bus, void *provider_data, 407 const struct attribute_group **groups, unsigned long flags, 408 unsigned long *dsm_mask); 409 410:: 411 412 /sys/devices/platform/nfit_test.0/ndbus0 413 |-- nmem0 414 | |-- available_slots 415 | |-- commands 416 | |-- dev 417 | |-- devtype 418 | |-- driver -> ../../../../../bus/nd/drivers/nvdimm 419 | |-- modalias 420 | |-- nfit 421 | | |-- device 422 | | |-- format 423 | | |-- handle 424 | | |-- phys_id 425 | | |-- rev_id 426 | | |-- serial 427 | | `-- vendor 428 | |-- state 429 | |-- subsystem -> ../../../../../bus/nd 430 | `-- uevent 431 |-- nmem1 432 [..] 433 434 435LIBNDCTL: DIMM enumeration example 436^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 437 438Note, in this example we are assuming NFIT-defined DIMMs which are 439identified by an "nfit_handle" a 32-bit value where: 440 441 - Bit 3:0 DIMM number within the memory channel 442 - Bit 7:4 memory channel number 443 - Bit 11:8 memory controller ID 444 - Bit 15:12 socket ID (within scope of a Node controller if node 445 controller is present) 446 - Bit 27:16 Node Controller ID 447 - Bit 31:28 Reserved 448 449:: 450 451 static struct ndctl_dimm *get_dimm_by_handle(struct ndctl_bus *bus, 452 unsigned int handle) 453 { 454 struct ndctl_dimm *dimm; 455 456 ndctl_dimm_foreach(bus, dimm) 457 if (ndctl_dimm_get_handle(dimm) == handle) 458 return dimm; 459 460 return NULL; 461 } 462 463 #define DIMM_HANDLE(n, s, i, c, d) \ 464 (((n & 0xfff) << 16) | ((s & 0xf) << 12) | ((i & 0xf) << 8) \ 465 | ((c & 0xf) << 4) | (d & 0xf)) 466 467 dimm = get_dimm_by_handle(bus, DIMM_HANDLE(0, 0, 0, 0, 0)); 468 469LIBNVDIMM/LIBNDCTL: Region 470-------------------------- 471 472A generic REGION device is registered for each PMEM range or BLK-aperture 473set. Per the example there are 6 regions: 2 PMEM and 4 BLK-aperture 474sets on the "nfit_test.0" bus. The primary role of regions are to be a 475container of "mappings". A mapping is a tuple of <DIMM, 476DPA-start-offset, length>. 477 478LIBNVDIMM provides a built-in driver for these REGION devices. This driver 479is responsible for reconciling the aliased DPA mappings across all 480regions, parsing the LABEL, if present, and then emitting NAMESPACE 481devices with the resolved/exclusive DPA-boundaries for the nd_pmem or 482nd_blk device driver to consume. 483 484In addition to the generic attributes of "mapping"s, "interleave_ways" 485and "size" the REGION device also exports some convenience attributes. 486"nstype" indicates the integer type of namespace-device this region 487emits, "devtype" duplicates the DEVTYPE variable stored by udev at the 488'add' event, "modalias" duplicates the MODALIAS variable stored by udev 489at the 'add' event, and finally, the optional "spa_index" is provided in 490the case where the region is defined by a SPA. 491 492LIBNVDIMM: region:: 493 494 struct nd_region *nvdimm_pmem_region_create(struct nvdimm_bus *nvdimm_bus, 495 struct nd_region_desc *ndr_desc); 496 struct nd_region *nvdimm_blk_region_create(struct nvdimm_bus *nvdimm_bus, 497 struct nd_region_desc *ndr_desc); 498 499:: 500 501 /sys/devices/platform/nfit_test.0/ndbus0 502 |-- region0 503 | |-- available_size 504 | |-- btt0 505 | |-- btt_seed 506 | |-- devtype 507 | |-- driver -> ../../../../../bus/nd/drivers/nd_region 508 | |-- init_namespaces 509 | |-- mapping0 510 | |-- mapping1 511 | |-- mappings 512 | |-- modalias 513 | |-- namespace0.0 514 | |-- namespace_seed 515 | |-- numa_node 516 | |-- nfit 517 | | `-- spa_index 518 | |-- nstype 519 | |-- set_cookie 520 | |-- size 521 | |-- subsystem -> ../../../../../bus/nd 522 | `-- uevent 523 |-- region1 524 [..] 525 526LIBNDCTL: region enumeration example 527^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 528 529Sample region retrieval routines based on NFIT-unique data like 530"spa_index" (interleave set id) for PMEM and "nfit_handle" (dimm id) for 531BLK:: 532 533 static struct ndctl_region *get_pmem_region_by_spa_index(struct ndctl_bus *bus, 534 unsigned int spa_index) 535 { 536 struct ndctl_region *region; 537 538 ndctl_region_foreach(bus, region) { 539 if (ndctl_region_get_type(region) != ND_DEVICE_REGION_PMEM) 540 continue; 541 if (ndctl_region_get_spa_index(region) == spa_index) 542 return region; 543 } 544 return NULL; 545 } 546 547 static struct ndctl_region *get_blk_region_by_dimm_handle(struct ndctl_bus *bus, 548 unsigned int handle) 549 { 550 struct ndctl_region *region; 551 552 ndctl_region_foreach(bus, region) { 553 struct ndctl_mapping *map; 554 555 if (ndctl_region_get_type(region) != ND_DEVICE_REGION_BLOCK) 556 continue; 557 ndctl_mapping_foreach(region, map) { 558 struct ndctl_dimm *dimm = ndctl_mapping_get_dimm(map); 559 560 if (ndctl_dimm_get_handle(dimm) == handle) 561 return region; 562 } 563 } 564 return NULL; 565 } 566 567 568Why Not Encode the Region Type into the Region Name? 569---------------------------------------------------- 570 571At first glance it seems since NFIT defines just PMEM and BLK interface 572types that we should simply name REGION devices with something derived 573from those type names. However, the ND subsystem explicitly keeps the 574REGION name generic and expects userspace to always consider the 575region-attributes for four reasons: 576 577 1. There are already more than two REGION and "namespace" types. For 578 PMEM there are two subtypes. As mentioned previously we have PMEM where 579 the constituent DIMM devices are known and anonymous PMEM. For BLK 580 regions the NFIT specification already anticipates vendor specific 581 implementations. The exact distinction of what a region contains is in 582 the region-attributes not the region-name or the region-devtype. 583 584 2. A region with zero child-namespaces is a possible configuration. For 585 example, the NFIT allows for a DCR to be published without a 586 corresponding BLK-aperture. This equates to a DIMM that can only accept 587 control/configuration messages, but no i/o through a descendant block 588 device. Again, this "type" is advertised in the attributes ('mappings' 589 == 0) and the name does not tell you much. 590 591 3. What if a third major interface type arises in the future? Outside 592 of vendor specific implementations, it's not difficult to envision a 593 third class of interface type beyond BLK and PMEM. With a generic name 594 for the REGION level of the device-hierarchy old userspace 595 implementations can still make sense of new kernel advertised 596 region-types. Userspace can always rely on the generic region 597 attributes like "mappings", "size", etc and the expected child devices 598 named "namespace". This generic format of the device-model hierarchy 599 allows the LIBNVDIMM and LIBNDCTL implementations to be more uniform and 600 future-proof. 601 602 4. There are more robust mechanisms for determining the major type of a 603 region than a device name. See the next section, How Do I Determine the 604 Major Type of a Region? 605 606How Do I Determine the Major Type of a Region? 607---------------------------------------------- 608 609Outside of the blanket recommendation of "use libndctl", or simply 610looking at the kernel header (/usr/include/linux/ndctl.h) to decode the 611"nstype" integer attribute, here are some other options. 612 6131. module alias lookup 614^^^^^^^^^^^^^^^^^^^^^^ 615 616 The whole point of region/namespace device type differentiation is to 617 decide which block-device driver will attach to a given LIBNVDIMM namespace. 618 One can simply use the modalias to lookup the resulting module. It's 619 important to note that this method is robust in the presence of a 620 vendor-specific driver down the road. If a vendor-specific 621 implementation wants to supplant the standard nd_blk driver it can with 622 minimal impact to the rest of LIBNVDIMM. 623 624 In fact, a vendor may also want to have a vendor-specific region-driver 625 (outside of nd_region). For example, if a vendor defined its own LABEL 626 format it would need its own region driver to parse that LABEL and emit 627 the resulting namespaces. The output from module resolution is more 628 accurate than a region-name or region-devtype. 629 6302. udev 631^^^^^^^ 632 633 The kernel "devtype" is registered in the udev database:: 634 635 # udevadm info --path=/devices/platform/nfit_test.0/ndbus0/region0 636 P: /devices/platform/nfit_test.0/ndbus0/region0 637 E: DEVPATH=/devices/platform/nfit_test.0/ndbus0/region0 638 E: DEVTYPE=nd_pmem 639 E: MODALIAS=nd:t2 640 E: SUBSYSTEM=nd 641 642 # udevadm info --path=/devices/platform/nfit_test.0/ndbus0/region4 643 P: /devices/platform/nfit_test.0/ndbus0/region4 644 E: DEVPATH=/devices/platform/nfit_test.0/ndbus0/region4 645 E: DEVTYPE=nd_blk 646 E: MODALIAS=nd:t3 647 E: SUBSYSTEM=nd 648 649 ...and is available as a region attribute, but keep in mind that the 650 "devtype" does not indicate sub-type variations and scripts should 651 really be understanding the other attributes. 652 6533. type specific attributes 654^^^^^^^^^^^^^^^^^^^^^^^^^^^ 655 656 As it currently stands a BLK-aperture region will never have a 657 "nfit/spa_index" attribute, but neither will a non-NFIT PMEM region. A 658 BLK region with a "mappings" value of 0 is, as mentioned above, a DIMM 659 that does not allow I/O. A PMEM region with a "mappings" value of zero 660 is a simple system-physical-address range. 661 662 663LIBNVDIMM/LIBNDCTL: Namespace 664----------------------------- 665 666A REGION, after resolving DPA aliasing and LABEL specified boundaries, 667surfaces one or more "namespace" devices. The arrival of a "namespace" 668device currently triggers either the nd_blk or nd_pmem driver to load 669and register a disk/block device. 670 671LIBNVDIMM: namespace 672^^^^^^^^^^^^^^^^^^^^ 673 674Here is a sample layout from the three major types of NAMESPACE where 675namespace0.0 represents DIMM-info-backed PMEM (note that it has a 'uuid' 676attribute), namespace2.0 represents a BLK namespace (note it has a 677'sector_size' attribute) that, and namespace6.0 represents an anonymous 678PMEM namespace (note that has no 'uuid' attribute due to not support a 679LABEL):: 680 681 /sys/devices/platform/nfit_test.0/ndbus0/region0/namespace0.0 682 |-- alt_name 683 |-- devtype 684 |-- dpa_extents 685 |-- force_raw 686 |-- modalias 687 |-- numa_node 688 |-- resource 689 |-- size 690 |-- subsystem -> ../../../../../../bus/nd 691 |-- type 692 |-- uevent 693 `-- uuid 694 /sys/devices/platform/nfit_test.0/ndbus0/region2/namespace2.0 695 |-- alt_name 696 |-- devtype 697 |-- dpa_extents 698 |-- force_raw 699 |-- modalias 700 |-- numa_node 701 |-- sector_size 702 |-- size 703 |-- subsystem -> ../../../../../../bus/nd 704 |-- type 705 |-- uevent 706 `-- uuid 707 /sys/devices/platform/nfit_test.1/ndbus1/region6/namespace6.0 708 |-- block 709 | `-- pmem0 710 |-- devtype 711 |-- driver -> ../../../../../../bus/nd/drivers/pmem 712 |-- force_raw 713 |-- modalias 714 |-- numa_node 715 |-- resource 716 |-- size 717 |-- subsystem -> ../../../../../../bus/nd 718 |-- type 719 `-- uevent 720 721LIBNDCTL: namespace enumeration example 722^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 723Namespaces are indexed relative to their parent region, example below. 724These indexes are mostly static from boot to boot, but subsystem makes 725no guarantees in this regard. For a static namespace identifier use its 726'uuid' attribute. 727 728:: 729 730 static struct ndctl_namespace 731 *get_namespace_by_id(struct ndctl_region *region, unsigned int id) 732 { 733 struct ndctl_namespace *ndns; 734 735 ndctl_namespace_foreach(region, ndns) 736 if (ndctl_namespace_get_id(ndns) == id) 737 return ndns; 738 739 return NULL; 740 } 741 742LIBNDCTL: namespace creation example 743^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 744 745Idle namespaces are automatically created by the kernel if a given 746region has enough available capacity to create a new namespace. 747Namespace instantiation involves finding an idle namespace and 748configuring it. For the most part the setting of namespace attributes 749can occur in any order, the only constraint is that 'uuid' must be set 750before 'size'. This enables the kernel to track DPA allocations 751internally with a static identifier:: 752 753 static int configure_namespace(struct ndctl_region *region, 754 struct ndctl_namespace *ndns, 755 struct namespace_parameters *parameters) 756 { 757 char devname[50]; 758 759 snprintf(devname, sizeof(devname), "namespace%d.%d", 760 ndctl_region_get_id(region), paramaters->id); 761 762 ndctl_namespace_set_alt_name(ndns, devname); 763 /* 'uuid' must be set prior to setting size! */ 764 ndctl_namespace_set_uuid(ndns, paramaters->uuid); 765 ndctl_namespace_set_size(ndns, paramaters->size); 766 /* unlike pmem namespaces, blk namespaces have a sector size */ 767 if (parameters->lbasize) 768 ndctl_namespace_set_sector_size(ndns, parameters->lbasize); 769 ndctl_namespace_enable(ndns); 770 } 771 772 773Why the Term "namespace"? 774^^^^^^^^^^^^^^^^^^^^^^^^^ 775 776 1. Why not "volume" for instance? "volume" ran the risk of confusing 777 ND (libnvdimm subsystem) to a volume manager like device-mapper. 778 779 2. The term originated to describe the sub-devices that can be created 780 within a NVME controller (see the nvme specification: 781 https://www.nvmexpress.org/specifications/), and NFIT namespaces are 782 meant to parallel the capabilities and configurability of 783 NVME-namespaces. 784 785 786LIBNVDIMM/LIBNDCTL: Block Translation Table "btt" 787------------------------------------------------- 788 789A BTT (design document: https://pmem.io/2014/09/23/btt.html) is a stacked 790block device driver that fronts either the whole block device or a 791partition of a block device emitted by either a PMEM or BLK NAMESPACE. 792 793LIBNVDIMM: btt layout 794^^^^^^^^^^^^^^^^^^^^^ 795 796Every region will start out with at least one BTT device which is the 797seed device. To activate it set the "namespace", "uuid", and 798"sector_size" attributes and then bind the device to the nd_pmem or 799nd_blk driver depending on the region type:: 800 801 /sys/devices/platform/nfit_test.1/ndbus0/region0/btt0/ 802 |-- namespace 803 |-- delete 804 |-- devtype 805 |-- modalias 806 |-- numa_node 807 |-- sector_size 808 |-- subsystem -> ../../../../../bus/nd 809 |-- uevent 810 `-- uuid 811 812LIBNDCTL: btt creation example 813^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 814 815Similar to namespaces an idle BTT device is automatically created per 816region. Each time this "seed" btt device is configured and enabled a new 817seed is created. Creating a BTT configuration involves two steps of 818finding and idle BTT and assigning it to consume a PMEM or BLK namespace:: 819 820 static struct ndctl_btt *get_idle_btt(struct ndctl_region *region) 821 { 822 struct ndctl_btt *btt; 823 824 ndctl_btt_foreach(region, btt) 825 if (!ndctl_btt_is_enabled(btt) 826 && !ndctl_btt_is_configured(btt)) 827 return btt; 828 829 return NULL; 830 } 831 832 static int configure_btt(struct ndctl_region *region, 833 struct btt_parameters *parameters) 834 { 835 btt = get_idle_btt(region); 836 837 ndctl_btt_set_uuid(btt, parameters->uuid); 838 ndctl_btt_set_sector_size(btt, parameters->sector_size); 839 ndctl_btt_set_namespace(btt, parameters->ndns); 840 /* turn off raw mode device */ 841 ndctl_namespace_disable(parameters->ndns); 842 /* turn on btt access */ 843 ndctl_btt_enable(btt); 844 } 845 846Once instantiated a new inactive btt seed device will appear underneath 847the region. 848 849Once a "namespace" is removed from a BTT that instance of the BTT device 850will be deleted or otherwise reset to default values. This deletion is 851only at the device model level. In order to destroy a BTT the "info 852block" needs to be destroyed. Note, that to destroy a BTT the media 853needs to be written in raw mode. By default, the kernel will autodetect 854the presence of a BTT and disable raw mode. This autodetect behavior 855can be suppressed by enabling raw mode for the namespace via the 856ndctl_namespace_set_raw_mode() API. 857 858 859Summary LIBNDCTL Diagram 860------------------------ 861 862For the given example above, here is the view of the objects as seen by the 863LIBNDCTL API:: 864 865 +---+ 866 |CTX| +---------+ +--------------+ +---------------+ 867 +-+-+ +-> REGION0 +---> NAMESPACE0.0 +--> PMEM8 "pm0.0" | 868 | | +---------+ +--------------+ +---------------+ 869 +-------+ | | +---------+ +--------------+ +---------------+ 870 | DIMM0 <-+ | +-> REGION1 +---> NAMESPACE1.0 +--> PMEM6 "pm1.0" | 871 +-------+ | | | +---------+ +--------------+ +---------------+ 872 | DIMM1 <-+ +-v--+ | +---------+ +--------------+ +---------------+ 873 +-------+ +-+BUS0+---> REGION2 +-+-> NAMESPACE2.0 +--> ND6 "blk2.0" | 874 | DIMM2 <-+ +----+ | +---------+ | +--------------+ +----------------------+ 875 +-------+ | | +-> NAMESPACE2.1 +--> ND5 "blk2.1" | BTT2 | 876 | DIMM3 <-+ | +--------------+ +----------------------+ 877 +-------+ | +---------+ +--------------+ +---------------+ 878 +-> REGION3 +-+-> NAMESPACE3.0 +--> ND4 "blk3.0" | 879 | +---------+ | +--------------+ +----------------------+ 880 | +-> NAMESPACE3.1 +--> ND3 "blk3.1" | BTT1 | 881 | +--------------+ +----------------------+ 882 | +---------+ +--------------+ +---------------+ 883 +-> REGION4 +---> NAMESPACE4.0 +--> ND2 "blk4.0" | 884 | +---------+ +--------------+ +---------------+ 885 | +---------+ +--------------+ +----------------------+ 886 +-> REGION5 +---> NAMESPACE5.0 +--> ND1 "blk5.0" | BTT0 | 887 +---------+ +--------------+ +---------------+------+ 888