.\"
.\" This file and its contents are supplied under the terms of the
.\" Common Development and Distribution License ("CDDL"), version 1.0.
.\" You may only use this file in accordance with the terms of version
.\" 1.0 of the CDDL.
.\"
.\" A full copy of the text of the CDDL should have accompanied this
.\" source.  A copy of the CDDL is also available via the Internet at
.\" http://www.illumos.org/license/CDDL.
.\"
.\"
.\" Copyright 2021 Oxide Computer Company
.\" Copyright 2022 Tintri by DDN, Inc. All rights reserved.
.\"
.Dd April 25, 2022
.Dt NVMEADM 8
.Os
.Sh NAME
.Nm nvmeadm
.Nd NVMe administration utility
.Sh SYNOPSIS
.Nm
.Fl h
.Op Ar command
.Nm
.Op Fl dv
.Cm list
.Oo
.Fl c
.Oc
.Oo
.Fl p o Ar field Ns [,...]
.Oc
.Op Ar ctl[/ns] Ns [,...]
.Nm
.Op Fl dv
.Cm identify
.Op Fl C | c | d | Oo Fl a Oc Fl n
.Ar ctl[/ns] Ns [,...]
.Nm
.Op Fl dv
.Cm identify-controller
.Op Fl C | c | Oo Fl a Oc Fl n
.Ar ctl Ns [,...]
.Nm
.Op Fl dv
.Cm identify-namespace
.Op Fl c | d
.Ar ctl/ns Ns [,...]
.Nm
.Op Fl dv
.Cm get-logpage
.Ar ctl[/ns] Ns [,...]
.Ar logpage
.Nm
.Op Fl dv
.Cm get-features
.Ar ctl[/ns] Ns [,...]
.Op Ar feature-list
.Nm
.Op Fl dv
.Cm format
.Ar ctl[/ns]
.Op Ar lba-format
.Nm
.Op Fl dv
.Cm secure-erase
.Op Fl c
.Ar ctl[/ns]
.Nm
.Op Fl dv
.Cm detach
.Ar ctl[/ns]
.Nm
.Op Fl dv
.Cm attach
.Ar ctl[/ns]
.Nm
.Op Fl dv
.Cm list-firmware
.Ar ctl
.Nm
.Op Fl dv
.Cm load-firmware
.Ar ctl
.Ar firmware-file
.Op Ar offset
.Nm
.Op Fl dv
.Cm commit-firmware
.Ar ctl
.Ar slot
.Nm
.Op Fl dv
.Cm activate-firmware
.Ar ctl
.Ar slot
.Sh DESCRIPTION
The
.Nm
utility can be used to enumerate the NVMe controllers and their
namespaces, query hardware information from a NVMe controller or
namespace, and to format or secure-erase a NVMe controller or
namespace.
.Pp
The information returned by the hardware is printed by
.Nm
in a human-readable form were applicable.
Generally all 0-based counts are normalized and values may be
converted to human-readable units such as MB (megabytes), W (watts),
or C (degrees Celsius).
.Sh OPTIONS
The following options are supported:
.Bl -tag -width Ds
.It Fl h
Print a short help text for
.Nm ,
or for an optionally specified
.Nm
command.
.It Fl d
Enable debugging output.
.It Fl v
Enable verbose output.
.El
.Sh ARGUMENTS
.Nm
expects the following kinds of arguments:
.Bl -tag -width "ctl[/ns]"
.It Ar command
Any command
.Nm
understands.
See section
.Sx COMMANDS .
.It Ar ctl[/ns]
Specifies a NVMe controller and optionally a namespace within that
controller.
The controller name consists of the driver name
.Qq nvme
followed by an instance number.
A namespace is specified by appending a single
.Qq /
to the controller name, followed by either the namespace ID or the namespace
EUI64 or NGUID as reported by the
.Cm identify
command.
The namespace ID is a positive non-zero decimal number.
For commands that don't change the device state multiple controllers
and namespaces can be specified as a comma-separated list.
.Pp
The list of controllers and namespaces present in the system can be
queried with the
.Cm list
command without any arguments.
.It Ar logpage
Specifies the log page name for the
.Cm get-logpage
command.
.It Ar feature-list
A comma-separated list of feature names for the
.Cm get-features
command.
Feature names can be specified in upper or lower case and can be
shortened the shortest unique name.
Some features may also have an alternative short name.
.It Ar lba-format
A non-zero integer specifying the LBA format for the
.Cm format
command.
The list of supported LBA formats on a namespace can be retrieved
with the
.Nm
.Cm identify
command.
.It Ar firmware-file
Specifies the name of a firmware file to be loaded into the controller
using the
.Cm load-firmware
command.
.It Ar offset
Specifies the byte offset at which to load
.Ar firmware-file
within the controller's upload buffer.
Vendors may require multiple images to be loaded at different offsets
before a firmware set is committed to a
.Ar slot .
.It Ar slot
Specifies the firmware slot into which a firmware set is committed
using the
.Cm commit-firmware
command, and subsequently activated with the
.Cm activate-firmware
command.
Slots and their contents can be printed using the
.Nm
.Cm list-firmware
command.
.El
.Sh COMMANDS
.Bl -tag -width ""
.It Xo
.Nm
.Cm list
.Oo
.Fl c
.Oc
.Oo
.Fl p o Ar field Ns [,...]
.Oc
.Op Ar ctl[/ns] Ns [,...]
.Xc
Lists the NVMe controllers and by default also their active
namespaces, printing a 1-line summary of their basic properties for each.
If a list of controllers and/or namespaces is given then the listing
is limited to those devices.
If no controllers or namespaces are given as arguments, then all controllers
in the system and their respective active namespaces are listed.
When using the
.Fl v
option
to
.Nm ,
all possible namespaces of the controllers will be listed.
.Pp
The
.Nm
.Cm list
command supports the following options:
.Bl -tag -width Fl
.It Fl c
List controllers only and not their namespaces.
.It Fl p
Produce parsable output rather than human-readable output.
This option requires that output fields be selected with the
.Fl o
option.
.It Fl o Ar field Ns [,...]
A comma-separated list of one or more output fields to be used.
Fields are listed below and the name is case insensitive.
.El
.Pp
The following fields can be specified when using the parsable form:
.Bl -tag -width UNALLOCATED
.It Sy MODEL
The model number of the device, generally containing information about
both the manufacturer and the product.
.It Sy SERIAL
The NVMe controller's serial number.
.It Sy FWREV
The controller's firmware revision.
.It Sy VERSION
The version of the NVMe specification the controller supports.
.It Sy INSTANCE
The name of the device node and instance of it.
.El
.Pp
In addition, the following fields can be specified when listing namespaces, not
using the
.Fl c
option:
.Bl -tag -width UNALLOCATED
.It Sy CAPACITY
The amount of logical bytes that the namespace may actually have allocated at
any time.
This may be different than size due to the use of thin provisioning or due to
administrative action.
.It Sy SIZE
The logical size in bytes of the namespace.
.It Sy USED
The number of bytes used in the namespace.
.It Sy NAMESPACE
The numerical value of the namespace which can be used as part of other
.Nm
operations.
.It Sy DISK
The name of the disk device that corresponds to the namespace, if any.
.El
.Pp
When using the
.Fl c
option to list controllers, the following additional fields are supported:
.Bl -tag -width UNALLOCATED
.It Sy CAPACITY
The total raw capacity of the NVMe controller in bytes.
.It Sy UNALLOCATED
The number of bytes not currently assigned to any namespace in the controller.
.El
.It Xo
.Nm
.Cm identify-controller
.Op Fl C | c | Oo Fl a Oc Fl n
.Ar ctl Ns [,...]
.Xc
Print detailed information about the specified controllers.
For an explanation of the data printed by this command refer to the description
of the
.Qq IDENTIFY
admin command in the NVMe specification.
.Pp
By default, a relevant subset of the
.Qq IDENTIFY CONTROLLER
data structure is printed.
The full data structure is only printed when verbose output is requested.
.Pp
The following options can be used to print other
.Qq IDENTIFY
information:
.Bl -tag -width Fl
.It Fl C
Print the Common Namespace Identification of the controller.
.It Fl a
Alter the output of the
.Fl n
option to print the list allocated namespace identifiers.
Can only be specified together with the
.Fl n
option.
.It Fl c
Print the list of all unique controller identifiers in the NVMe subsystem the
specified controller belongs to.
.It Fl n
Print the list of active namespace identifiers of the controller.
.El
.It Xo
.Nm
.Cm identify-namespace
.Op Fl c | d
.Ar ctl/ns Ns [,...]
.Xc
Print detailed information about the specified namespace.
For an explanation of the data printed by this command refer to the description
of the
.Qq IDENTIFY
admin command in the NVMe specification.
.Pp
By default, a relevant subset of the
.Qq IDENTIFY NAMESPACE
data structure is printed.
The full data structure is only printed when verbose output is requested.
.Pp
The following options can be used to print other
.Qq IDENTIFY
information:
.Bl -tag -width Fl
.It Fl c
Print the list of all unique controller identifiers in the NVMe subsystem the
specified namespace belongs to and which are currently attached to this
namespace.
.It Fl d
Print the list of namespace identification descriptors of the namespace.
.El
.It Xo
.Nm
.Cm identify
.Op Fl C | c | d | Oo Fl a Oc Fl n
.Ar ctl[/ns] Ns [,...]
.Xc
Short-hand for the
.Cm identify-controller
and
.Cm identify-namespace
commands, prints the same information about the specified controllers and/or
namespaces, depending on whether a controller or a namespace was specified.
.Pp
For a description of the various optional flags refer to the above description
of the
.Cm identify-controller
and
.Cm identify-namespace
commands.
.It Xo
.Nm
.Cm get-logpage
.Ar ctl[/ns] Ns [,...]
.Ar logpage
.Xc
Print the specified log page of the specified controllers and/or namespaces.
Most log pages are only available on a per-controller basis.
Known log pages are:
.Bl -tag -width "firmware"
.It error
Error Information
.It health
SMART/Health Information.
A controller may support this log page on a per-namespace basis.
.It firmware
Firmware Slot Information
.El
.Pp
For an explanation of the contents of the log pages refer to the
description of the
.Qq GET LOGPAGE
admin command in the NVMe specification.
.It Xo
.Nm
.Cm get-features
.Ar ctl[/ns] Ns [,...]
.Op Ar feature-list
.Xc
Prints information about the specified features, or all features if
none are given, of the specified controllers and/or namespaces.
Feature names are case-insensitive, and they can be shortened as long
as they remain unique.
Some features also have alternative short names to which the same
rules apply.
The following features are supported:
.Pp
.TS
tab(:);
l l l.
FULL NAME:SHORT NAME:CONTROLLER/NAMESPACE
Arbitration::controller
Power Management::controller
LBA Range Type:range:namespace
Temperature Threshold::controller
Error Recovery::controller
Volatile Write Cache:cache:controller
Number of Queues:queues:controller
Interrupt Coalescing:coalescing:controller
Interrupt Vector Configuration:vector:controller
Write Atomicity:atomicity:controller
Asynchronous Event Configuration:event:controller
Autonomous Power State Transition::controller
Software Progress Marker:progress:controller
.TE
.Pp
For an explanation of the individual features refer to the description
of the
.Qq SET FEATURES
admin command in the NVMe specification.
.It Xo
.Nm
.Cm format
.Ar ctl[/ns]
.Op Ar lba-format
.Xc
Formats the specified namespace or all namespaces of the specified
controller.
This command implies a
.Nm
.Cm detach
and subsequent
.Nm
.Cm attach
of the specified namespace(s), which will cause a changed LBA format
to be detected.
If no LBA format is specified the LBA format currently used by the
namespace will be used.
When formatting all namespaces without specifying a LBA format the LBA
format of namespace 1 will be used.
A list of LBA formats supported by a namespace can be queried with the
.Nm
.Cm identify
command.
.Pp
Note that not all devices support formatting individual or all
namespaces, or support formatting at all.
.Pp
LBA formats using a non-zero metadata size are not supported by
.Nm
or
.Xr nvme 4D .
.Pp
The list of supported LBA formats on a namespace can be retrieved
with the
.Nm
.Cm identify
command.
.It Xo
.Nm
.Cm secure-erase
.Op Fl c
.Ar ctl[/ns]
.Xc
Erases the specified namespace or all namespaces of the controller.
The flag
.Fl c
will cause a cryptographic erase instead of a normal erase.
This command implies a
.Nm
.Cm detach
and
.Nm
.Cm attach
of the specified namespace(s).
.Pp
Note that not all devices support erasing individual or all
namespaces, or support erasing at all.
.It Xo
.Nm
.Cm detach
.Ar ctl[/ns]
.Xc
Temporarily detaches the
.Xr blkdev 4D
instance from the specified namespace or all namespaces of the controller.
This will prevent I/O access to the affected namespace(s).
Detach will only succeed if the affected namespace(s) are not
currently opened.
The detached state will not persist across reboots or reloads of the
.Xr nvme 4D
driver.
.Pp
It is not an error to detach a namespace that is already detached, any such
request will be silently ignored.
.It Xo
.Nm
.Cm attach
.Ar ctl[/ns]
.Xc
Attaches the
.Xr blkdev 4D
instance to the specified namespace or all namespaces of the controller.
This will make I/O accesses to the namespace(s) possible again after a
previous
.Nm
.Cm detach
command.
.Pp
It is not an error to attach a namespace that is already attached, any such
request will be silently ignored.
.It Xo
.Nm
.Cm list-firmware
.Ar ctl
.Xc
List currently active firmware slot, the next active firmware slot, and the
current contents of all firmware slots of an NVMe controller.
This is a synonym for the
.Nm
.Cm get-logpage
.Ar ctl
.Cm firmware
command.
.It Xo
.Nm
.Cm load-firmware
.Ar ctl
.Ar firmware-file
.Op Ar offset
.Xc
Loads
.Ar firmware-file
into the controller's upload memory at
.Ar offset ,
the default is 0. A vendor may require multiple files to be loaded
at different offsets before the firmware is committed to a
.Ar slot .
.It Xo
.Nm
.Cm commit-firmware
.Ar ctl
.Ar slot
.Xc
Commits firmware previously loaded by the
.Cm load-firmware
command to
.Ar slot .
.It Xo
.Nm
.Cm activate-firmware
.Ar ctl
.Ar slot
.Xc
Activates the firmware in slot
.Ar slot .
The firmware image in
.Ar slot
is activated at the next NVM controller reset.
.El
.Sh EXIT STATUS
.Ex -std
.Sh EXAMPLES
.Bl -tag -width ""
.It Sy Example 1: List all NVMe controllers and namespaces
.Bd -literal
# nvmeadm list
nvme1: model: INTEL SSDPEDMD800G4, serial: CVFT4134001R800CGN, FW rev: 8DV10049, NVMe v1.0
  nvme1/1 (c1t1d0): Size = 763097 MB, Capacity = 763097 MB, Used = 763097 MB
nvme4: model: SAMSUNG MZVPV128HDGM-00000, serial: S1XVNYAGA00640, FW rev: BXW7300Q, NVMe v1.1
  nvme4/1 (c2t2d0): Size = 122104 MB, Capacity = 122104 MB, Used = 5127 MB
.Ed
.It Sy Example 2: Identify a namespace
.Bd -literal
# nvmeadm identify nvme4/1
nvme4/1: Identify Namespace
  Namespace Capabilities and Features
    Namespace Size:                         122104MB
    Namespace Capacity:                     122104MB
    Namespace Utilization:                  5127MB
    Namespace Features
      Thin Provisioning:                    unsupported
    Number of LBA Formats:                  1
    Formatted LBA Size
      LBA Format:                           1
      Extended Data LBA:                    no
    Metadata Capabilities
      Extended Data LBA:                    unsupported
      Separate Metadata:                    unsupported
    End-to-End Data Protection Capabilities
      Protection Information Type 1:        unsupported
      Protection Information Type 2:        unsupported
      Protection Information Type 3:        unsupported
      Protection Information first:         unsupported
      Protection Information last:          unsupported
    End-to-End Data Protection Settings
      Protection Information:               disabled
      Protection Information in Metadata:   last 8 bytes
    LBA Format 1
      Metadata Size:                        0 bytes
      LBA Data Size:                        512 bytes
      Relative Performance:                 Best
.Ed
.It Sy Example 3: Get SMART/Health information (verbose)
.Bd -literal
# nvmeadm -v get-logpage nvme4/1 health
nvme4/1: SMART/Health Information
  Critical Warnings
    Available Space:                        OK
    Temperature:                            OK
    Device Reliability:                     OK
    Media:                                  OK
    Volatile Memory Backup:                 OK
  Temperature:                              37C
  Available Spare Capacity:                 100%
  Available Spare Threshold:                10%
  Device Life Used:                         0%
  Data Read:                                0GB
  Data Written:                             64GB
  Read Commands:                            52907
  Write Commands:                           567874
  Controller Busy:                          1min
  Power Cycles:                             6
  Power On:                                 141h
  Unsafe Shutdowns:                         1
  Uncorrectable Media Errors:               0
  Errors Logged:                            1
.Ed
.It Sy Example 4: Get Asynchronous Event Configuration information
.Bd -literal
# nvmeadm get-features nvme0,nvme4 event,power
nvme0: Get Features
  Asynchronous Event Configuration
    Available Space below threshold:        disabled
    Temperature above threshold:            disabled
    Device Reliability compromised:         disabled
    Media read-only:                        disabled
  Power Management
    Power State:                            0
nvme4: Get Features
  Asynchronous Event Configuration
    Available Space below threshold:        disabled
    Temperature above threshold:            disabled
    Device Reliability compromised:         disabled
    Media read-only:                        disabled
    Volatile Memory Backup failed:          disabled
  Power Management
    Power State:                            0
.Ed
.It Sy Example 5: Load and activate firmware
.Bd -literal
# nvmeadm list-firmware nvme3
nvme3: Firmware Slot Information
  Active Firmware Slot:                     4
  Next Firmware Slot:                       4
  Firmware Revision for Slot 1:             KNGND110 (read-only)
  Firmware Revision for Slot 2:             KNGND110
  Firmware Revision for Slot 3:             KNGND110
  Firmware Revision for Slot 4:             KNGND112
  Firmware Revision for Slot 5:             KNGND110

# nvmeadm -v load-firmware nvme3 KNGND113.bin
1740544 bytes downloaded.

# nvmeadm -v commit-firmware nvme3 5
Firmware committed to slot 5.

# nvmeadm -v activate-firmware nvme3 5
Slot 5 activated: NVM subsystem reset required - power cycle your system.

# nvmeadm list-firmware nvme3
nvme3: Firmware Slot Information
  Active Firmware Slot:                     4
  Next Firmware Slot:                       5
  Firmware Revision for Slot 1:             KNGND110 (read-only)
  Firmware Revision for Slot 2:             KNGND110
  Firmware Revision for Slot 3:             KNGND110
  Firmware Revision for Slot 4:             KNGND112
  Firmware Revision for Slot 5:             KNGND113
.Ed
.El
.Sh INTERFACE STABILITY
The command line interface of
.Nm
is
.Sy Evolving .
The output of
.Nm
is
.Sy Not-an-Interface
and may change any time.
.Sh SEE ALSO
.Xr nvme 4D
.Pp
.Lk http://www.nvmexpress.org/specifications/ "NVMe specifications"