1=============== 2RDMA Controller 3=============== 4 5.. Contents 6 7 1. Overview 8 1-1. What is RDMA controller? 9 1-2. Why RDMA controller needed? 10 1-3. How is RDMA controller implemented? 11 2. Usage Examples 12 3. RDMA Interface Files 13 141. Overview 15=========== 16 171-1. What is RDMA controller? 18----------------------------- 19 20RDMA controller allows user to limit RDMA/IB specific resources that a given 21set of processes can use. These processes are grouped using RDMA controller. 22 23RDMA controller defines two resources which can be limited for processes of a 24cgroup. 25 261-2. Why RDMA controller needed? 27-------------------------------- 28 29Currently user space applications can easily take away all the rdma verb 30specific resources such as AH, CQ, QP, MR etc. Due to which other applications 31in other cgroup or kernel space ULPs may not even get chance to allocate any 32rdma resources. This can lead to service unavailability. 33 34Therefore RDMA controller is needed through which resource consumption 35of processes can be limited. Through this controller different rdma 36resources can be accounted. 37 381-3. How is RDMA controller implemented? 39---------------------------------------- 40 41RDMA cgroup allows limit configuration of resources. Rdma cgroup maintains 42resource accounting per cgroup, per device using resource pool structure. 43Each such resource pool is limited up to 64 resources in given resource pool 44by rdma cgroup, which can be extended later if required. 45 46This resource pool object is linked to the cgroup css. Typically there 47are 0 to 4 resource pool instances per cgroup, per device in most use cases. 48But nothing limits to have it more. At present hundreds of RDMA devices per 49single cgroup may not be handled optimally, however there is no 50known use case or requirement for such configuration either. 51 52Since RDMA resources can be allocated from any process and can be freed by any 53of the child processes which shares the address space, rdma resources are 54always owned by the creator cgroup css. This allows process migration from one 55to other cgroup without major complexity of transferring resource ownership; 56because such ownership is not really present due to shared nature of 57rdma resources. Linking resources around css also ensures that cgroups can be 58deleted after processes migrated. This allow progress migration as well with 59active resources, even though that is not a primary use case. 60 61Whenever RDMA resource charging occurs, owner rdma cgroup is returned to 62the caller. Same rdma cgroup should be passed while uncharging the resource. 63This also allows process migrated with active RDMA resource to charge 64to new owner cgroup for new resource. It also allows to uncharge resource of 65a process from previously charged cgroup which is migrated to new cgroup, 66even though that is not a primary use case. 67 68Resource pool object is created in following situations. 69(a) User sets the limit and no previous resource pool exist for the device 70of interest for the cgroup. 71(b) No resource limits were configured, but IB/RDMA stack tries to 72charge the resource. So that it correctly uncharge them when applications are 73running without limits and later on when limits are enforced during uncharging, 74otherwise usage count will drop to negative. 75 76Resource pool is destroyed if all the resource limits are set to max and 77it is the last resource getting deallocated. 78 79User should set all the limit to max value if it intents to remove/unconfigure 80the resource pool for a particular device. 81 82IB stack honors limits enforced by the rdma controller. When application 83query about maximum resource limits of IB device, it returns minimum of 84what is configured by user for a given cgroup and what is supported by 85IB device. 86 87Following resources can be accounted by rdma controller. 88 89 ========== ============================= 90 hca_handle Maximum number of HCA Handles 91 hca_object Maximum number of HCA Objects 92 ========== ============================= 93 942. Usage Examples 95================= 96 97(a) Configure resource limit:: 98 99 echo mlx4_0 hca_handle=2 hca_object=2000 > /sys/fs/cgroup/rdma/1/rdma.max 100 echo ocrdma1 hca_handle=3 > /sys/fs/cgroup/rdma/2/rdma.max 101 102(b) Query resource limit:: 103 104 cat /sys/fs/cgroup/rdma/2/rdma.max 105 #Output: 106 mlx4_0 hca_handle=2 hca_object=2000 107 ocrdma1 hca_handle=3 hca_object=max 108 109(c) Query current usage:: 110 111 cat /sys/fs/cgroup/rdma/2/rdma.current 112 #Output: 113 mlx4_0 hca_handle=1 hca_object=20 114 ocrdma1 hca_handle=1 hca_object=23 115 116(d) Delete resource limit:: 117 118 echo mlx4_0 hca_handle=max hca_object=max > /sys/fs/cgroup/rdma/1/rdma.max 119 1203. RDMA Interface Files 121======================== 122 123The following interface files are available in each non-root RDMA cgroup. 124 125 rdma.max 126 A read-write file which describes the configured resource limit 127 for an RDMA/IB device. See the Usage Examples above. 128 129 rdma.current 130 A read-only file which describes the current resource usage. 131 132 rdma.peak 133 A read-only nested-keyed file which shows the historical high 134 watermark of resource usage per device since the cgroup was created. 135 136 An example for mlx4 and ocrdma device follows:: 137 138 mlx4_0 hca_handle=1 hca_object=20 139 ocrdma1 hca_handle=0 hca_object=23 140 141 rdma.events 142 A read-only nested-keyed file which exists on non-root cgroups 143 and contains the following keys: 144 145 max 146 The number of times a process in this cgroup or its 147 descendants attempted an RDMA resource allocation that 148 was rejected because a rdma.max limit in the subtree 149 was reached. This is a hierarchical counter propagated 150 upward to all ancestor cgroups. A value change in this 151 file generates a file modified event. 152 153 alloc_fail 154 The number of RDMA resource allocation attempts that 155 originated in this cgroup or its descendants and failed 156 due to a rdma.max limit being reached. This is a 157 hierarchical counter propagated upward. 158 159 An example for mlx4 device follows:: 160 161 mlx4_0 hca_handle.max=5 hca_handle.alloc_fail=3 hca_object.max=0 hca_object.alloc_fail=0 162 163 rdma.events.local 164 Similar to rdma.events but the fields are local to the cgroup, 165 i.e. not hierarchical. The file modified event generated on this 166 file reflects only the local events. 167 168 The following nested keys are defined. 169 170 max 171 The number of times a process in this cgroup or its 172 descendants attempted an RDMA resource allocation that 173 was rejected because this cgroup's own rdma.max limit 174 was reached. 175 176 alloc_fail 177 The number of RDMA resource allocation attempts 178 originating from this cgroup that failed due to this 179 cgroup's or an ancestor's rdma.max limit. 180 181 An example for mlx4 device follows:: 182 183 mlx4_0 hca_handle.max=5 hca_handle.alloc_fail=0 hca_object.max=0 hca_object.alloc_fail=0 184