1=============== 2RDMA Controller 3=============== 4 5.. Contents 6 7 1. Overview 8 1-1. What is RDMA controller? 9 1-2. Why RDMA controller needed? 10 1-3. How is RDMA controller implemented? 11 2. Usage Examples 12 3. RDMA Interface Files 13 141. Overview 15=========== 16 171-1. What is RDMA controller? 18----------------------------- 19 20RDMA controller allows user to limit RDMA/IB specific resources that a given 21set of processes can use. These processes are grouped using RDMA controller. 22 23RDMA controller defines two resources which can be limited for processes of a 24cgroup. 25 261-2. Why RDMA controller needed? 27-------------------------------- 28 29Currently user space applications can easily take away all the rdma verb 30specific resources such as AH, CQ, QP, MR etc. Due to which other applications 31in other cgroup or kernel space ULPs may not even get chance to allocate any 32rdma resources. This can lead to service unavailability. 33 34Therefore RDMA controller is needed through which resource consumption 35of processes can be limited. Through this controller different rdma 36resources can be accounted. 37 381-3. How is RDMA controller implemented? 39---------------------------------------- 40 41RDMA cgroup allows limit configuration of resources. Rdma cgroup maintains 42resource accounting per cgroup, per device using resource pool structure. 43Each such resource pool is limited up to 64 resources in given resource pool 44by rdma cgroup, which can be extended later if required. 45 46This resource pool object is linked to the cgroup css. Typically there 47are 0 to 4 resource pool instances per cgroup, per device in most use cases. 48But nothing limits to have it more. At present hundreds of RDMA devices per 49single cgroup may not be handled optimally, however there is no 50known use case or requirement for such configuration either. 51 52Since RDMA resources can be allocated from any process and can be freed by any 53of the child processes which shares the address space, rdma resources are 54always owned by the creator cgroup css. This allows process migration from one 55to other cgroup without major complexity of transferring resource ownership; 56because such ownership is not really present due to shared nature of 57rdma resources. Linking resources around css also ensures that cgroups can be 58deleted after processes migrated. This allow progress migration as well with 59active resources, even though that is not a primary use case. 60 61Whenever RDMA resource charging occurs, owner rdma cgroup is returned to 62the caller. Same rdma cgroup should be passed while uncharging the resource. 63This also allows process migrated with active RDMA resource to charge 64to new owner cgroup for new resource. It also allows to uncharge resource of 65a process from previously charged cgroup which is migrated to new cgroup, 66even though that is not a primary use case. 67 68Resource pool object is created in following situations. 69(a) User sets the limit and no previous resource pool exist for the device 70of interest for the cgroup. 71(b) No resource limits were configured, but IB/RDMA stack tries to 72charge the resource. So that it correctly uncharge them when applications are 73running without limits and later on when limits are enforced during uncharging, 74otherwise usage count will drop to negative. 75 76Resource pool is destroyed if all the resource limits are set to max and 77it is the last resource getting deallocated. 78 79User should set all the limit to max value if it intents to remove/unconfigure 80the resource pool for a particular device. 81 82IB stack honors limits enforced by the rdma controller. When application 83query about maximum resource limits of IB device, it returns minimum of 84what is configured by user for a given cgroup and what is supported by 85IB device. 86 87Following resources can be accounted by rdma controller. 88 89 ========== ============================= 90 hca_handle Maximum number of HCA Handles 91 hca_object Maximum number of HCA Objects 92 ========== ============================= 93 94RDMA devices from all network namespaces are listed. Each line starts with 95the device name. If more than one device has the same name, ``index=N`` 96follows the name, where ``N`` is the system-wide RDMA device index, unique 97among registered devices. When configuring a limit, the index is optional 98for a globally unique name and required for a duplicate name. A write without 99the required index fails with ``-ENOTUNIQ``. 100 1012. Usage Examples 102================= 103 104(a) Configure resource limit:: 105 106 echo mlx4_0 hca_handle=2 hca_object=2000 > /sys/fs/cgroup/rdma/1/rdma.max 107 echo ocrdma1 hca_handle=3 > /sys/fs/cgroup/rdma/2/rdma.max 108 echo "rxe0 index=5 hca_handle=2" > /sys/fs/cgroup/rdma/3/rdma.max 109 110(b) Query resource limit:: 111 112 cat /sys/fs/cgroup/rdma/2/rdma.max 113 #Output: 114 mlx4_0 hca_handle=2 hca_object=2000 115 ocrdma1 hca_handle=3 hca_object=max 116 117(c) Query current usage:: 118 119 cat /sys/fs/cgroup/rdma/2/rdma.current 120 #Output: 121 mlx4_0 hca_handle=1 hca_object=20 122 ocrdma1 hca_handle=1 hca_object=23 123 124(d) Delete resource limit:: 125 126 echo mlx4_0 hca_handle=max hca_object=max > /sys/fs/cgroup/rdma/1/rdma.max 127 1283. RDMA Interface Files 129======================== 130 131The following interface files are available in each non-root RDMA cgroup. 132 133 rdma.max 134 A read-write file which describes the configured resource limit 135 for an RDMA/IB device. See the Usage Examples above. 136 137 rdma.current 138 A read-only file which describes the current resource usage. 139 140 rdma.peak 141 A read-only nested-keyed file which shows the historical high 142 watermark of resource usage per device since the cgroup was created. 143 144 An example for mlx4 and ocrdma device follows:: 145 146 mlx4_0 hca_handle=1 hca_object=20 147 ocrdma1 hca_handle=0 hca_object=23 148 149 rdma.events 150 A read-only nested-keyed file which exists on non-root cgroups 151 and contains the following keys: 152 153 max 154 The number of times a process in this cgroup or its 155 descendants attempted an RDMA resource allocation that 156 was rejected because a rdma.max limit in the subtree 157 was reached. This is a hierarchical counter propagated 158 upward to all ancestor cgroups. A value change in this 159 file generates a file modified event. 160 161 alloc_fail 162 The number of RDMA resource allocation attempts that 163 originated in this cgroup or its descendants and failed 164 due to a rdma.max limit being reached. This is a 165 hierarchical counter propagated upward. 166 167 An example for mlx4 device follows:: 168 169 mlx4_0 hca_handle.max=5 hca_handle.alloc_fail=3 hca_object.max=0 hca_object.alloc_fail=0 170 171 rdma.events.local 172 Similar to rdma.events but the fields are local to the cgroup, 173 i.e. not hierarchical. The file modified event generated on this 174 file reflects only the local events. 175 176 The following nested keys are defined. 177 178 max 179 The number of times a process in this cgroup or its 180 descendants attempted an RDMA resource allocation that 181 was rejected because this cgroup's own rdma.max limit 182 was reached. 183 184 alloc_fail 185 The number of RDMA resource allocation attempts 186 originating from this cgroup that failed due to this 187 cgroup's or an ancestor's rdma.max limit. 188 189 An example for mlx4 device follows:: 190 191 mlx4_0 hca_handle.max=5 hca_handle.alloc_fail=0 hca_object.max=0 hca_object.alloc_fail=0 192