en Copyright 2015 Java http://kernelsources.org:8080/source/history/freebsd/sys/modules/nvmf/Makefile#a15f7c96a27644de3ed3bfcf5feee285ebc1cc91 nvmft: The in-kernel NVMe over Fabrics controllerThis is the server (target in SCSI terms) for NVMe over Fabrics.Userland is responsible for accepting a new queue pair and receivingthe initial Connect command before handing the queue pair off via anioctl to this CTL frontend.This frontend exposes CTL LUNs as NVMe namespaces to remote hosts.Users can ask LUNS to CTL that can be shared via either iSCSI orNVMeoF.Reviewed by: impSponsored by: Chelsio CommunicationsDifferential Revision: https://reviews.freebsd.org/D44726 List of files: /freebsd/sys/modules/nvmf/Makefile Fri, 03 May 2024 01:34:45 +0200 John Baldwin <jhb@FreeBSD.org> http://kernelsources.org:8080/source/history/freebsd/sys/modules/nvmf/Makefile#a1eda74167b5edb99fd31d507d8a3f7d7e14ae2b nvmf: The in-kernel NVMe over Fabrics hostThis is the client (initiator in SCSI terms) for NVMe over Fabrics.Userland is responsible for creating a set of queue pairs and thenhanding them off via an ioctl to this driver, e.g. via the 'connect'command from nvmecontrol(8). An nvmeX new-bus device is createdat the top-level to represent the remote controller similar to PCInvmeX devices for PCI-express controllers.As with nvme(4), namespace devices named /dev/nvmeXnsY are created andpass through commands can be submitted to either the namespace devicesor the controller device. For example, 'nvmecontrol identify nvmeX'works for a remote Fabrics controller the same as for a PCI-expresscontroller.nvmf exports remote namespaces via nda(4) devices using the new NVMFCAM transport. nvmf does not support nvd(4), only nda(4).Sponsored by: Chelsio CommunicationsDifferential Revision: https://reviews.freebsd.org/D44714 List of files: /freebsd/sys/modules/nvmf/Makefile Fri, 03 May 2024 01:29:37 +0200 John Baldwin <jhb@FreeBSD.org> http://kernelsources.org:8080/source/history/freebsd/sys/modules/nvmf/Makefile#59144db3fca192c4637637dfe6b5a5d98632cd47 nvmf_tcp: Add a TCP transport for NVMe over FabricsStructurally this is very similar to the TCP transport for iSCSI(icl_soft.c). One key difference is that NVMeoF transports use a moreabstract interface working with NVMe commands rather than transportPDUs. Thus, the data transfer for a given command is managed entirelyin the transport backend.Similar to icl_soft.c, separate kthreads are used to handle transmitand receive for each queue pair. On the transmit side, when a capsuleis transmitted by an upper layer, it is placed on a queue forprocessing by the transmit thread. The transmit thread convertscommand response capsules into suitable TCP PDUs where each PDU isdescribed by an mbuf chain that is then queued to the backing socket'ssend buffer. Command capsules can embed data along with the NVMecommand.On the receive side, a socket upcall notifies the receive kthread whenmore data arrives. Once enough data has arrived for a PDU, the PDU ishandled synchronously in the kthread. PDUs such as R2T or datarelated PDUs are handled internally, with callbacks invoked if a datatransfer encounters an error, or once the data transfer has completed.Received capsule PDUs invoke the upper layer's capsule_receivedcallback.struct nvmf_tcp_command_buffer manages a TCP command buffer for datatransfers that do not use in-capsule-data as described in the NVMeoFspec. Data related PDUs such as R2T, C2H, and H2C are associated witha command buffer except in the case of the send_controller_datatransport method which simply constructs one or more C2H PDUs from thecaller's mbuf chain.Sponsored by: Chelsio CommunicationsDifferential Revision: https://reviews.freebsd.org/D44712 List of files: /freebsd/sys/modules/nvmf/Makefile Fri, 03 May 2024 01:28:47 +0200 John Baldwin <jhb@FreeBSD.org> http://kernelsources.org:8080/source/history/freebsd/sys/modules/nvmf/Makefile#aa1207ea4f030c50a91bca6a3df950ca25113d5a nvmf: Add infrastructure kernel module for NVMe over Fabricsnvmf_transport.ko provides routines for managing NVMeoF queue pairsand capsules. It provides a glue layer between transports (such asTCP or RDMA) and an NVMeoF host (initiator) and controller (target).Unlike the synchronous API exposed to the host and controller bylibnvmf, the kernel's transport layer uses an asynchronous API builton callbacks. Upper layers provide callbacks on queue pairs that areinvoked for transport errors (error_cb) or anytime a capsule isreceived (receive_cb).Data transfers for a command are usually associated with a callbackthat is invoked once a transfer has finished either due to an erroror successful completion.For an upper layer that is a host, command capsules are allocated andpopulated with an NVMe SQE by calling nvmf_allocate_command. A databuffer (described by a struct memdesc) can be associated with acommand capsule before it is transmitted via nvmf_capsule_append_data.This function accepts a direction (send vs receive) as well as thedata transfer callback. The host then transmits the command vianvmf_transmit_capsule. The host must ensure that the data bufferdescribed by the 'struct memdesc' remains valid until the datatransfer callback is called. The queue pair's receive_cb callbackshould match received response capsules up with previously transmittedcommands.For the controller, incoming commands are received via the queuepair's receive_cb callback. nvmf_receive_controller_data is used toretrieve any data from a command (e.g. the data for a WRITE command).It can be called multiple times to split the data transfer intosmaller sizes. This function accepts an I/O completion callback thatis invoked once the data transfer has completed.nvmf_send_controller_data is used to send data to a remote host inresponse to a command. In this case a callback function is not usedbut the status is returned synchronously. Finally, the controller canallocate a response capsule via nvmf_allocate_response populated witha supplied CQE and send the response via nvmf_transmit_capsule.Reviewed by: impSponsored by: Chelsio CommunicationsDifferential Revision: https://reviews.freebsd.org/D44711 List of files: /freebsd/sys/modules/nvmf/Makefile Fri, 03 May 2024 01:28:32 +0200 John Baldwin <jhb@FreeBSD.org>