1*1c6bbc45SMeetakshi Setiya.. SPDX-License-Identifier: GPL-2.0 2*1c6bbc45SMeetakshi Setiya 3*1c6bbc45SMeetakshi Setiya=========================== 4*1c6bbc45SMeetakshi SetiyaSMB Direct - SMB3 over RDMA 5*1c6bbc45SMeetakshi Setiya=========================== 6*1c6bbc45SMeetakshi Setiya 7*1c6bbc45SMeetakshi SetiyaThis document describes how to set up the Linux SMB client and server to 8*1c6bbc45SMeetakshi Setiyause RDMA. 9*1c6bbc45SMeetakshi Setiya 10*1c6bbc45SMeetakshi SetiyaOverview 11*1c6bbc45SMeetakshi Setiya======== 12*1c6bbc45SMeetakshi SetiyaThe Linux SMB kernel client supports SMB Direct, which is a transport 13*1c6bbc45SMeetakshi Setiyascheme for SMB3 that uses RDMA (Remote Direct Memory Access) to provide 14*1c6bbc45SMeetakshi Setiyahigh throughput and low latencies by bypassing the traditional TCP/IP 15*1c6bbc45SMeetakshi Setiyastack. 16*1c6bbc45SMeetakshi SetiyaSMB Direct on the Linux SMB client can be tested against KSMBD - a 17*1c6bbc45SMeetakshi Setiyakernel-space SMB server. 18*1c6bbc45SMeetakshi Setiya 19*1c6bbc45SMeetakshi SetiyaInstallation 20*1c6bbc45SMeetakshi Setiya============= 21*1c6bbc45SMeetakshi Setiya- Install an RDMA device. As long as the RDMA device driver is supported 22*1c6bbc45SMeetakshi Setiya by the kernel, it should work. This includes both software emulators (soft 23*1c6bbc45SMeetakshi Setiya RoCE, soft iWARP) and hardware devices (InfiniBand, RoCE, iWARP). 24*1c6bbc45SMeetakshi Setiya 25*1c6bbc45SMeetakshi Setiya- Install a kernel with SMB Direct support. The first kernel release to 26*1c6bbc45SMeetakshi Setiya support SMB Direct on both the client and server side is 5.15. Therefore, 27*1c6bbc45SMeetakshi Setiya a distribution compatible with kernel 5.15 or later is required. 28*1c6bbc45SMeetakshi Setiya 29*1c6bbc45SMeetakshi Setiya- Install cifs-utils, which provides the `mount.cifs` command to mount SMB 30*1c6bbc45SMeetakshi Setiya shares. 31*1c6bbc45SMeetakshi Setiya 32*1c6bbc45SMeetakshi Setiya- Configure the RDMA stack 33*1c6bbc45SMeetakshi Setiya 34*1c6bbc45SMeetakshi Setiya Make sure that your kernel configuration has RDMA support enabled. Under 35*1c6bbc45SMeetakshi Setiya Device Drivers -> Infiniband support, update the kernel configuration to 36*1c6bbc45SMeetakshi Setiya enable Infiniband support. 37*1c6bbc45SMeetakshi Setiya 38*1c6bbc45SMeetakshi Setiya Enable the appropriate IB HCA support or iWARP adapter support, 39*1c6bbc45SMeetakshi Setiya depending on your hardware. 40*1c6bbc45SMeetakshi Setiya 41*1c6bbc45SMeetakshi Setiya If you are using InfiniBand, enable IP-over-InfiniBand support. 42*1c6bbc45SMeetakshi Setiya 43*1c6bbc45SMeetakshi Setiya For soft RDMA, enable either the soft iWARP (`RDMA _SIW`) or soft RoCE 44*1c6bbc45SMeetakshi Setiya (`RDMA_RXE`) module. Install the `iproute2` package and use the 45*1c6bbc45SMeetakshi Setiya `rdma link add` command to load the module and create an 46*1c6bbc45SMeetakshi Setiya RDMA interface. 47*1c6bbc45SMeetakshi Setiya 48*1c6bbc45SMeetakshi Setiya e.g. if your local ethernet interface is `eth0`, you can use: 49*1c6bbc45SMeetakshi Setiya 50*1c6bbc45SMeetakshi Setiya .. code-block:: bash 51*1c6bbc45SMeetakshi Setiya 52*1c6bbc45SMeetakshi Setiya sudo rdma link add siw0 type siw netdev eth0 53*1c6bbc45SMeetakshi Setiya 54*1c6bbc45SMeetakshi Setiya- Enable SMB Direct support for both the server and the client in the kernel 55*1c6bbc45SMeetakshi Setiya configuration. 56*1c6bbc45SMeetakshi Setiya 57*1c6bbc45SMeetakshi Setiya Server Setup 58*1c6bbc45SMeetakshi Setiya 59*1c6bbc45SMeetakshi Setiya .. code-block:: text 60*1c6bbc45SMeetakshi Setiya 61*1c6bbc45SMeetakshi Setiya Network File Systems ---> 62*1c6bbc45SMeetakshi Setiya <M> SMB3 server support 63*1c6bbc45SMeetakshi Setiya [*] Support for SMB Direct protocol 64*1c6bbc45SMeetakshi Setiya 65*1c6bbc45SMeetakshi Setiya Client Setup 66*1c6bbc45SMeetakshi Setiya 67*1c6bbc45SMeetakshi Setiya .. code-block:: text 68*1c6bbc45SMeetakshi Setiya 69*1c6bbc45SMeetakshi Setiya Network File Systems ---> 70*1c6bbc45SMeetakshi Setiya <M> SMB3 and CIFS support (advanced network filesystem) 71*1c6bbc45SMeetakshi Setiya [*] SMB Direct support 72*1c6bbc45SMeetakshi Setiya 73*1c6bbc45SMeetakshi Setiya- Build and install the kernel. SMB Direct support will be enabled in the 74*1c6bbc45SMeetakshi Setiya cifs.ko and ksmbd.ko modules. 75*1c6bbc45SMeetakshi Setiya 76*1c6bbc45SMeetakshi SetiyaSetup and Usage 77*1c6bbc45SMeetakshi Setiya================ 78*1c6bbc45SMeetakshi Setiya 79*1c6bbc45SMeetakshi Setiya- Set up and start a KSMBD server as described in the `KSMBD documentation 80*1c6bbc45SMeetakshi Setiya <https://www.kernel.org/doc/Documentation/filesystems/smb/ksmbd.rst>`_. 81*1c6bbc45SMeetakshi Setiya Also add the "server multi channel support = yes" parameter to ksmbd.conf. 82*1c6bbc45SMeetakshi Setiya 83*1c6bbc45SMeetakshi Setiya- On the client, mount the share with `rdma` mount option to use SMB Direct 84*1c6bbc45SMeetakshi Setiya (specify a SMB version 3.0 or higher using `vers`). 85*1c6bbc45SMeetakshi Setiya 86*1c6bbc45SMeetakshi Setiya For example: 87*1c6bbc45SMeetakshi Setiya 88*1c6bbc45SMeetakshi Setiya .. code-block:: bash 89*1c6bbc45SMeetakshi Setiya 90*1c6bbc45SMeetakshi Setiya mount -t cifs //server/share /mnt/point -o vers=3.1.1,rdma 91*1c6bbc45SMeetakshi Setiya 92*1c6bbc45SMeetakshi Setiya- To verify that the mount is using SMB Direct, you can check dmesg for the 93*1c6bbc45SMeetakshi Setiya following log line after mounting: 94*1c6bbc45SMeetakshi Setiya 95*1c6bbc45SMeetakshi Setiya .. code-block:: text 96*1c6bbc45SMeetakshi Setiya 97*1c6bbc45SMeetakshi Setiya CIFS: VFS: RDMA transport established 98*1c6bbc45SMeetakshi Setiya 99*1c6bbc45SMeetakshi Setiya Or, verify `rdma` mount option for the share in `/proc/mounts`: 100*1c6bbc45SMeetakshi Setiya 101*1c6bbc45SMeetakshi Setiya .. code-block:: bash 102*1c6bbc45SMeetakshi Setiya 103*1c6bbc45SMeetakshi Setiya cat /proc/mounts | grep cifs 104