1aa4d16e4SJack Wang******************************** 2aa4d16e4SJack WangRDMA Network Block Device (RNBD) 3aa4d16e4SJack Wang******************************** 4aa4d16e4SJack Wang 5aa4d16e4SJack WangIntroduction 6aa4d16e4SJack Wang------------ 7aa4d16e4SJack Wang 8aa4d16e4SJack WangRNBD (RDMA Network Block Device) is a pair of kernel modules 9aa4d16e4SJack Wang(client and server) that allow for remote access of a block device on 10aa4d16e4SJack Wangthe server over RTRS protocol using the RDMA (InfiniBand, RoCE, iWARP) 11aa4d16e4SJack Wangtransport. After being mapped, the remote block devices can be accessed 12aa4d16e4SJack Wangon the client side as local block devices. 13aa4d16e4SJack Wang 14aa4d16e4SJack WangI/O is transferred between client and server by the RTRS transport 15aa4d16e4SJack Wangmodules. The administration of RNBD and RTRS modules is done via 16aa4d16e4SJack Wangsysfs entries. 17aa4d16e4SJack Wang 18aa4d16e4SJack WangRequirements 19aa4d16e4SJack Wang------------ 20aa4d16e4SJack Wang 21aa4d16e4SJack Wang RTRS kernel modules 22aa4d16e4SJack Wang 23aa4d16e4SJack WangQuick Start 24aa4d16e4SJack Wang----------- 25aa4d16e4SJack Wang 26aa4d16e4SJack WangServer side: 27aa4d16e4SJack Wang # modprobe rnbd_server 28aa4d16e4SJack Wang 29aa4d16e4SJack WangClient side: 30aa4d16e4SJack Wang # modprobe rnbd_client 31aa4d16e4SJack Wang # echo "sessname=blya path=ip:10.50.100.66 device_path=/dev/ram0" > \ 32aa4d16e4SJack Wang /sys/devices/virtual/rnbd-client/ctl/map_device 33aa4d16e4SJack Wang 34aa4d16e4SJack Wang Where "sessname=" is a session name, a string to identify the session 35aa4d16e4SJack Wang on client and on server sides; "path=" is a destination IP address or 36aa4d16e4SJack Wang a pair of a source and a destination IPs, separated by comma. Multiple 37aa4d16e4SJack Wang "path=" options can be specified in order to use multipath (see RTRS 38aa4d16e4SJack Wang description for details); "device_path=" is the block device to be 39aa4d16e4SJack Wang mapped from the server side. After the session to the server machine is 40aa4d16e4SJack Wang established, the mapped device will appear on the client side under 41aa4d16e4SJack Wang /dev/rnbd<N>. 42aa4d16e4SJack Wang 43aa4d16e4SJack Wang 44aa4d16e4SJack WangRNBD-Server Module Parameters 45aa4d16e4SJack Wang============================= 46aa4d16e4SJack Wang 47aa4d16e4SJack Wangdev_search_path 48aa4d16e4SJack Wang--------------- 49aa4d16e4SJack Wang 50aa4d16e4SJack WangWhen a device is mapped from the client, the server generates the path 51aa4d16e4SJack Wangto the block device on the server side by concatenating dev_search_path 52aa4d16e4SJack Wangand the "device_path" that was specified in the map_device operation. 53aa4d16e4SJack Wang 54aa4d16e4SJack WangThe default dev_search_path is: "/". 55aa4d16e4SJack Wang 56aa4d16e4SJack Wangdev_search_path option can also contain %SESSNAME% in order to provide 57aa4d16e4SJack Wangdifferent device namespaces for different sessions. See "device_path" 58aa4d16e4SJack Wangoption for details. 59aa4d16e4SJack Wang 60aa4d16e4SJack Wang============================ 61aa4d16e4SJack WangProtocol (rnbd/rnbd-proto.h) 62aa4d16e4SJack Wang============================ 63aa4d16e4SJack Wang 64aa4d16e4SJack Wang1. Before mapping first device from a given server, client sends an 65aa4d16e4SJack WangRNBD_MSG_SESS_INFO to the server. Server responds with 66aa4d16e4SJack WangRNBD_MSG_SESS_INFO_RSP. Currently the messages only contain the protocol 67aa4d16e4SJack Wangversion for backward compatibility. 68aa4d16e4SJack Wang 69aa4d16e4SJack Wang2. Client requests to open a device by sending RNBD_MSG_OPEN message. This 70aa4d16e4SJack Wangcontains the path to the device and access mode (read-only or writable). 71aa4d16e4SJack WangServer responds to the message with RNBD_MSG_OPEN_RSP. This contains 72aa4d16e4SJack Wanga 32 bit device id to be used for IOs and device "geometry" related 73aa4d16e4SJack Wanginformation: side, max_hw_sectors, etc. 74aa4d16e4SJack Wang 75aa4d16e4SJack Wang3. Client attaches RNBD_MSG_IO to each IO message send to a device. This 76aa4d16e4SJack Wangmessage contains device id, provided by server in his rnbd_msg_open_rsp, 77aa4d16e4SJack Wangsector to be accessed, read-write flags and bi_size. 78aa4d16e4SJack Wang 79aa4d16e4SJack Wang4. Client closes a device by sending RNBD_MSG_CLOSE which contains only the 80aa4d16e4SJack Wangdevice id provided by the server. 81aa4d16e4SJack Wang 82aa4d16e4SJack Wang========================================= 83aa4d16e4SJack WangContributors List(in alphabetical order) 84aa4d16e4SJack Wang========================================= 85aa4d16e4SJack WangDanil Kipnis <danil.kipnis@profitbricks.com> 86aa4d16e4SJack WangFabian Holler <mail@fholler.de> 87aa4d16e4SJack WangGuoqing Jiang <guoqing.jiang@cloud.ionos.com> 88aa4d16e4SJack WangJack Wang <jinpu.wang@profitbricks.com> 89aa4d16e4SJack WangKleber Souza <kleber.souza@profitbricks.com> 90aa4d16e4SJack WangLutz Pogrell <lutz.pogrell@cloud.ionos.com> 91aa4d16e4SJack WangMilind Dumbare <Milind.dumbare@gmail.com> 92aa4d16e4SJack WangRoman Penyaev <roman.penyaev@profitbricks.com> 93*ef8048ddSSwapnil IngleSwapnil Ingle <ingleswapnil@gmail.com> 94