1*b9dd2beaSMauro Carvalho Chehab.. SPDX-License-Identifier: GPL-2.0 2*b9dd2beaSMauro Carvalho Chehab 3*b9dd2beaSMauro Carvalho Chehab============================= 4*b9dd2beaSMauro Carvalho ChehabKernel Connection Multiplexor 5*b9dd2beaSMauro Carvalho Chehab============================= 6*b9dd2beaSMauro Carvalho Chehab 7*b9dd2beaSMauro Carvalho ChehabKernel Connection Multiplexor (KCM) is a mechanism that provides a message based 8*b9dd2beaSMauro Carvalho Chehabinterface over TCP for generic application protocols. With KCM an application 9*b9dd2beaSMauro Carvalho Chehabcan efficiently send and receive application protocol messages over TCP using 10*b9dd2beaSMauro Carvalho Chehabdatagram sockets. 11*b9dd2beaSMauro Carvalho Chehab 12*b9dd2beaSMauro Carvalho ChehabKCM implements an NxM multiplexor in the kernel as diagrammed below:: 13*b9dd2beaSMauro Carvalho Chehab 14*b9dd2beaSMauro Carvalho Chehab +------------+ +------------+ +------------+ +------------+ 15*b9dd2beaSMauro Carvalho Chehab | KCM socket | | KCM socket | | KCM socket | | KCM socket | 16*b9dd2beaSMauro Carvalho Chehab +------------+ +------------+ +------------+ +------------+ 17*b9dd2beaSMauro Carvalho Chehab | | | | 18*b9dd2beaSMauro Carvalho Chehab +-----------+ | | +----------+ 19*b9dd2beaSMauro Carvalho Chehab | | | | 20*b9dd2beaSMauro Carvalho Chehab +----------------------------------+ 21*b9dd2beaSMauro Carvalho Chehab | Multiplexor | 22*b9dd2beaSMauro Carvalho Chehab +----------------------------------+ 23*b9dd2beaSMauro Carvalho Chehab | | | | | 24*b9dd2beaSMauro Carvalho Chehab +---------+ | | | ------------+ 25*b9dd2beaSMauro Carvalho Chehab | | | | | 26*b9dd2beaSMauro Carvalho Chehab +----------+ +----------+ +----------+ +----------+ +----------+ 27*b9dd2beaSMauro Carvalho Chehab | Psock | | Psock | | Psock | | Psock | | Psock | 28*b9dd2beaSMauro Carvalho Chehab +----------+ +----------+ +----------+ +----------+ +----------+ 29*b9dd2beaSMauro Carvalho Chehab | | | | | 30*b9dd2beaSMauro Carvalho Chehab +----------+ +----------+ +----------+ +----------+ +----------+ 31*b9dd2beaSMauro Carvalho Chehab | TCP sock | | TCP sock | | TCP sock | | TCP sock | | TCP sock | 32*b9dd2beaSMauro Carvalho Chehab +----------+ +----------+ +----------+ +----------+ +----------+ 33*b9dd2beaSMauro Carvalho Chehab 34*b9dd2beaSMauro Carvalho ChehabKCM sockets 35*b9dd2beaSMauro Carvalho Chehab=========== 36*b9dd2beaSMauro Carvalho Chehab 37*b9dd2beaSMauro Carvalho ChehabThe KCM sockets provide the user interface to the multiplexor. All the KCM sockets 38*b9dd2beaSMauro Carvalho Chehabbound to a multiplexor are considered to have equivalent function, and I/O 39*b9dd2beaSMauro Carvalho Chehaboperations in different sockets may be done in parallel without the need for 40*b9dd2beaSMauro Carvalho Chehabsynchronization between threads in userspace. 41*b9dd2beaSMauro Carvalho Chehab 42*b9dd2beaSMauro Carvalho ChehabMultiplexor 43*b9dd2beaSMauro Carvalho Chehab=========== 44*b9dd2beaSMauro Carvalho Chehab 45*b9dd2beaSMauro Carvalho ChehabThe multiplexor provides the message steering. In the transmit path, messages 46*b9dd2beaSMauro Carvalho Chehabwritten on a KCM socket are sent atomically on an appropriate TCP socket. 47*b9dd2beaSMauro Carvalho ChehabSimilarly, in the receive path, messages are constructed on each TCP socket 48*b9dd2beaSMauro Carvalho Chehab(Psock) and complete messages are steered to a KCM socket. 49*b9dd2beaSMauro Carvalho Chehab 50*b9dd2beaSMauro Carvalho ChehabTCP sockets & Psocks 51*b9dd2beaSMauro Carvalho Chehab==================== 52*b9dd2beaSMauro Carvalho Chehab 53*b9dd2beaSMauro Carvalho ChehabTCP sockets may be bound to a KCM multiplexor. A Psock structure is allocated 54*b9dd2beaSMauro Carvalho Chehabfor each bound TCP socket, this structure holds the state for constructing 55*b9dd2beaSMauro Carvalho Chehabmessages on receive as well as other connection specific information for KCM. 56*b9dd2beaSMauro Carvalho Chehab 57*b9dd2beaSMauro Carvalho ChehabConnected mode semantics 58*b9dd2beaSMauro Carvalho Chehab======================== 59*b9dd2beaSMauro Carvalho Chehab 60*b9dd2beaSMauro Carvalho ChehabEach multiplexor assumes that all attached TCP connections are to the same 61*b9dd2beaSMauro Carvalho Chehabdestination and can use the different connections for load balancing when 62*b9dd2beaSMauro Carvalho Chehabtransmitting. The normal send and recv calls (include sendmmsg and recvmmsg) 63*b9dd2beaSMauro Carvalho Chehabcan be used to send and receive messages from the KCM socket. 64*b9dd2beaSMauro Carvalho Chehab 65*b9dd2beaSMauro Carvalho ChehabSocket types 66*b9dd2beaSMauro Carvalho Chehab============ 67*b9dd2beaSMauro Carvalho Chehab 68*b9dd2beaSMauro Carvalho ChehabKCM supports SOCK_DGRAM and SOCK_SEQPACKET socket types. 69*b9dd2beaSMauro Carvalho Chehab 70*b9dd2beaSMauro Carvalho ChehabMessage delineation 71*b9dd2beaSMauro Carvalho Chehab------------------- 72*b9dd2beaSMauro Carvalho Chehab 73*b9dd2beaSMauro Carvalho ChehabMessages are sent over a TCP stream with some application protocol message 74*b9dd2beaSMauro Carvalho Chehabformat that typically includes a header which frames the messages. The length 75*b9dd2beaSMauro Carvalho Chehabof a received message can be deduced from the application protocol header 76*b9dd2beaSMauro Carvalho Chehab(often just a simple length field). 77*b9dd2beaSMauro Carvalho Chehab 78*b9dd2beaSMauro Carvalho ChehabA TCP stream must be parsed to determine message boundaries. Berkeley Packet 79*b9dd2beaSMauro Carvalho ChehabFilter (BPF) is used for this. When attaching a TCP socket to a multiplexor a 80*b9dd2beaSMauro Carvalho ChehabBPF program must be specified. The program is called at the start of receiving 81*b9dd2beaSMauro Carvalho Chehaba new message and is given an skbuff that contains the bytes received so far. 82*b9dd2beaSMauro Carvalho ChehabIt parses the message header and returns the length of the message. Given this 83*b9dd2beaSMauro Carvalho Chehabinformation, KCM will construct the message of the stated length and deliver it 84*b9dd2beaSMauro Carvalho Chehabto a KCM socket. 85*b9dd2beaSMauro Carvalho Chehab 86*b9dd2beaSMauro Carvalho ChehabTCP socket management 87*b9dd2beaSMauro Carvalho Chehab--------------------- 88*b9dd2beaSMauro Carvalho Chehab 89*b9dd2beaSMauro Carvalho ChehabWhen a TCP socket is attached to a KCM multiplexor data ready (POLLIN) and 90*b9dd2beaSMauro Carvalho Chehabwrite space available (POLLOUT) events are handled by the multiplexor. If there 91*b9dd2beaSMauro Carvalho Chehabis a state change (disconnection) or other error on a TCP socket, an error is 92*b9dd2beaSMauro Carvalho Chehabposted on the TCP socket so that a POLLERR event happens and KCM discontinues 93*b9dd2beaSMauro Carvalho Chehabusing the socket. When the application gets the error notification for a 94*b9dd2beaSMauro Carvalho ChehabTCP socket, it should unattach the socket from KCM and then handle the error 95*b9dd2beaSMauro Carvalho Chehabcondition (the typical response is to close the socket and create a new 96*b9dd2beaSMauro Carvalho Chehabconnection if necessary). 97*b9dd2beaSMauro Carvalho Chehab 98*b9dd2beaSMauro Carvalho ChehabKCM limits the maximum receive message size to be the size of the receive 99*b9dd2beaSMauro Carvalho Chehabsocket buffer on the attached TCP socket (the socket buffer size can be set by 100*b9dd2beaSMauro Carvalho ChehabSO_RCVBUF). If the length of a new message reported by the BPF program is 101*b9dd2beaSMauro Carvalho Chehabgreater than this limit a corresponding error (EMSGSIZE) is posted on the TCP 102*b9dd2beaSMauro Carvalho Chehabsocket. The BPF program may also enforce a maximum messages size and report an 103*b9dd2beaSMauro Carvalho Chehaberror when it is exceeded. 104*b9dd2beaSMauro Carvalho Chehab 105*b9dd2beaSMauro Carvalho ChehabA timeout may be set for assembling messages on a receive socket. The timeout 106*b9dd2beaSMauro Carvalho Chehabvalue is taken from the receive timeout of the attached TCP socket (this is set 107*b9dd2beaSMauro Carvalho Chehabby SO_RCVTIMEO). If the timer expires before assembly is complete an error 108*b9dd2beaSMauro Carvalho Chehab(ETIMEDOUT) is posted on the socket. 109*b9dd2beaSMauro Carvalho Chehab 110*b9dd2beaSMauro Carvalho ChehabUser interface 111*b9dd2beaSMauro Carvalho Chehab============== 112*b9dd2beaSMauro Carvalho Chehab 113*b9dd2beaSMauro Carvalho ChehabCreating a multiplexor 114*b9dd2beaSMauro Carvalho Chehab---------------------- 115*b9dd2beaSMauro Carvalho Chehab 116*b9dd2beaSMauro Carvalho ChehabA new multiplexor and initial KCM socket is created by a socket call:: 117*b9dd2beaSMauro Carvalho Chehab 118*b9dd2beaSMauro Carvalho Chehab socket(AF_KCM, type, protocol) 119*b9dd2beaSMauro Carvalho Chehab 120*b9dd2beaSMauro Carvalho Chehab- type is either SOCK_DGRAM or SOCK_SEQPACKET 121*b9dd2beaSMauro Carvalho Chehab- protocol is KCMPROTO_CONNECTED 122*b9dd2beaSMauro Carvalho Chehab 123*b9dd2beaSMauro Carvalho ChehabCloning KCM sockets 124*b9dd2beaSMauro Carvalho Chehab------------------- 125*b9dd2beaSMauro Carvalho Chehab 126*b9dd2beaSMauro Carvalho ChehabAfter the first KCM socket is created using the socket call as described 127*b9dd2beaSMauro Carvalho Chehababove, additional sockets for the multiplexor can be created by cloning 128*b9dd2beaSMauro Carvalho Chehaba KCM socket. This is accomplished by an ioctl on a KCM socket:: 129*b9dd2beaSMauro Carvalho Chehab 130*b9dd2beaSMauro Carvalho Chehab /* From linux/kcm.h */ 131*b9dd2beaSMauro Carvalho Chehab struct kcm_clone { 132*b9dd2beaSMauro Carvalho Chehab int fd; 133*b9dd2beaSMauro Carvalho Chehab }; 134*b9dd2beaSMauro Carvalho Chehab 135*b9dd2beaSMauro Carvalho Chehab struct kcm_clone info; 136*b9dd2beaSMauro Carvalho Chehab 137*b9dd2beaSMauro Carvalho Chehab memset(&info, 0, sizeof(info)); 138*b9dd2beaSMauro Carvalho Chehab 139*b9dd2beaSMauro Carvalho Chehab err = ioctl(kcmfd, SIOCKCMCLONE, &info); 140*b9dd2beaSMauro Carvalho Chehab 141*b9dd2beaSMauro Carvalho Chehab if (!err) 142*b9dd2beaSMauro Carvalho Chehab newkcmfd = info.fd; 143*b9dd2beaSMauro Carvalho Chehab 144*b9dd2beaSMauro Carvalho ChehabAttach transport sockets 145*b9dd2beaSMauro Carvalho Chehab------------------------ 146*b9dd2beaSMauro Carvalho Chehab 147*b9dd2beaSMauro Carvalho ChehabAttaching of transport sockets to a multiplexor is performed by calling an 148*b9dd2beaSMauro Carvalho Chehabioctl on a KCM socket for the multiplexor. e.g.:: 149*b9dd2beaSMauro Carvalho Chehab 150*b9dd2beaSMauro Carvalho Chehab /* From linux/kcm.h */ 151*b9dd2beaSMauro Carvalho Chehab struct kcm_attach { 152*b9dd2beaSMauro Carvalho Chehab int fd; 153*b9dd2beaSMauro Carvalho Chehab int bpf_fd; 154*b9dd2beaSMauro Carvalho Chehab }; 155*b9dd2beaSMauro Carvalho Chehab 156*b9dd2beaSMauro Carvalho Chehab struct kcm_attach info; 157*b9dd2beaSMauro Carvalho Chehab 158*b9dd2beaSMauro Carvalho Chehab memset(&info, 0, sizeof(info)); 159*b9dd2beaSMauro Carvalho Chehab 160*b9dd2beaSMauro Carvalho Chehab info.fd = tcpfd; 161*b9dd2beaSMauro Carvalho Chehab info.bpf_fd = bpf_prog_fd; 162*b9dd2beaSMauro Carvalho Chehab 163*b9dd2beaSMauro Carvalho Chehab ioctl(kcmfd, SIOCKCMATTACH, &info); 164*b9dd2beaSMauro Carvalho Chehab 165*b9dd2beaSMauro Carvalho ChehabThe kcm_attach structure contains: 166*b9dd2beaSMauro Carvalho Chehab 167*b9dd2beaSMauro Carvalho Chehab - fd: file descriptor for TCP socket being attached 168*b9dd2beaSMauro Carvalho Chehab - bpf_prog_fd: file descriptor for compiled BPF program downloaded 169*b9dd2beaSMauro Carvalho Chehab 170*b9dd2beaSMauro Carvalho ChehabUnattach transport sockets 171*b9dd2beaSMauro Carvalho Chehab-------------------------- 172*b9dd2beaSMauro Carvalho Chehab 173*b9dd2beaSMauro Carvalho ChehabUnattaching a transport socket from a multiplexor is straightforward. An 174*b9dd2beaSMauro Carvalho Chehab"unattach" ioctl is done with the kcm_unattach structure as the argument:: 175*b9dd2beaSMauro Carvalho Chehab 176*b9dd2beaSMauro Carvalho Chehab /* From linux/kcm.h */ 177*b9dd2beaSMauro Carvalho Chehab struct kcm_unattach { 178*b9dd2beaSMauro Carvalho Chehab int fd; 179*b9dd2beaSMauro Carvalho Chehab }; 180*b9dd2beaSMauro Carvalho Chehab 181*b9dd2beaSMauro Carvalho Chehab struct kcm_unattach info; 182*b9dd2beaSMauro Carvalho Chehab 183*b9dd2beaSMauro Carvalho Chehab memset(&info, 0, sizeof(info)); 184*b9dd2beaSMauro Carvalho Chehab 185*b9dd2beaSMauro Carvalho Chehab info.fd = cfd; 186*b9dd2beaSMauro Carvalho Chehab 187*b9dd2beaSMauro Carvalho Chehab ioctl(fd, SIOCKCMUNATTACH, &info); 188*b9dd2beaSMauro Carvalho Chehab 189*b9dd2beaSMauro Carvalho ChehabDisabling receive on KCM socket 190*b9dd2beaSMauro Carvalho Chehab------------------------------- 191*b9dd2beaSMauro Carvalho Chehab 192*b9dd2beaSMauro Carvalho ChehabA setsockopt is used to disable or enable receiving on a KCM socket. 193*b9dd2beaSMauro Carvalho ChehabWhen receive is disabled, any pending messages in the socket's 194*b9dd2beaSMauro Carvalho Chehabreceive buffer are moved to other sockets. This feature is useful 195*b9dd2beaSMauro Carvalho Chehabif an application thread knows that it will be doing a lot of 196*b9dd2beaSMauro Carvalho Chehabwork on a request and won't be able to service new messages for a 197*b9dd2beaSMauro Carvalho Chehabwhile. Example use:: 198*b9dd2beaSMauro Carvalho Chehab 199*b9dd2beaSMauro Carvalho Chehab int val = 1; 200*b9dd2beaSMauro Carvalho Chehab 201*b9dd2beaSMauro Carvalho Chehab setsockopt(kcmfd, SOL_KCM, KCM_RECV_DISABLE, &val, sizeof(val)) 202*b9dd2beaSMauro Carvalho Chehab 203*b9dd2beaSMauro Carvalho ChehabBFP programs for message delineation 204*b9dd2beaSMauro Carvalho Chehab------------------------------------ 205*b9dd2beaSMauro Carvalho Chehab 206*b9dd2beaSMauro Carvalho ChehabBPF programs can be compiled using the BPF LLVM backend. For example, 207*b9dd2beaSMauro Carvalho Chehabthe BPF program for parsing Thrift is:: 208*b9dd2beaSMauro Carvalho Chehab 209*b9dd2beaSMauro Carvalho Chehab #include "bpf.h" /* for __sk_buff */ 210*b9dd2beaSMauro Carvalho Chehab #include "bpf_helpers.h" /* for load_word intrinsic */ 211*b9dd2beaSMauro Carvalho Chehab 212*b9dd2beaSMauro Carvalho Chehab SEC("socket_kcm") 213*b9dd2beaSMauro Carvalho Chehab int bpf_prog1(struct __sk_buff *skb) 214*b9dd2beaSMauro Carvalho Chehab { 215*b9dd2beaSMauro Carvalho Chehab return load_word(skb, 0) + 4; 216*b9dd2beaSMauro Carvalho Chehab } 217*b9dd2beaSMauro Carvalho Chehab 218*b9dd2beaSMauro Carvalho Chehab char _license[] SEC("license") = "GPL"; 219*b9dd2beaSMauro Carvalho Chehab 220*b9dd2beaSMauro Carvalho ChehabUse in applications 221*b9dd2beaSMauro Carvalho Chehab=================== 222*b9dd2beaSMauro Carvalho Chehab 223*b9dd2beaSMauro Carvalho ChehabKCM accelerates application layer protocols. Specifically, it allows 224*b9dd2beaSMauro Carvalho Chehabapplications to use a message based interface for sending and receiving 225*b9dd2beaSMauro Carvalho Chehabmessages. The kernel provides necessary assurances that messages are sent 226*b9dd2beaSMauro Carvalho Chehaband received atomically. This relieves much of the burden applications have 227*b9dd2beaSMauro Carvalho Chehabin mapping a message based protocol onto the TCP stream. KCM also make 228*b9dd2beaSMauro Carvalho Chehabapplication layer messages a unit of work in the kernel for the purposes of 229*b9dd2beaSMauro Carvalho Chehabsteering and scheduling, which in turn allows a simpler networking model in 230*b9dd2beaSMauro Carvalho Chehabmultithreaded applications. 231*b9dd2beaSMauro Carvalho Chehab 232*b9dd2beaSMauro Carvalho ChehabConfigurations 233*b9dd2beaSMauro Carvalho Chehab-------------- 234*b9dd2beaSMauro Carvalho Chehab 235*b9dd2beaSMauro Carvalho ChehabIn an Nx1 configuration, KCM logically provides multiple socket handles 236*b9dd2beaSMauro Carvalho Chehabto the same TCP connection. This allows parallelism between in I/O 237*b9dd2beaSMauro Carvalho Chehaboperations on the TCP socket (for instance copyin and copyout of data is 238*b9dd2beaSMauro Carvalho Chehabparallelized). In an application, a KCM socket can be opened for each 239*b9dd2beaSMauro Carvalho Chehabprocessing thread and inserted into the epoll (similar to how SO_REUSEPORT 240*b9dd2beaSMauro Carvalho Chehabis used to allow multiple listener sockets on the same port). 241*b9dd2beaSMauro Carvalho Chehab 242*b9dd2beaSMauro Carvalho ChehabIn a MxN configuration, multiple connections are established to the 243*b9dd2beaSMauro Carvalho Chehabsame destination. These are used for simple load balancing. 244*b9dd2beaSMauro Carvalho Chehab 245*b9dd2beaSMauro Carvalho ChehabMessage batching 246*b9dd2beaSMauro Carvalho Chehab---------------- 247*b9dd2beaSMauro Carvalho Chehab 248*b9dd2beaSMauro Carvalho ChehabThe primary purpose of KCM is load balancing between KCM sockets and hence 249*b9dd2beaSMauro Carvalho Chehabthreads in a nominal use case. Perfect load balancing, that is steering 250*b9dd2beaSMauro Carvalho Chehabeach received message to a different KCM socket or steering each sent 251*b9dd2beaSMauro Carvalho Chehabmessage to a different TCP socket, can negatively impact performance 252*b9dd2beaSMauro Carvalho Chehabsince this doesn't allow for affinities to be established. Balancing 253*b9dd2beaSMauro Carvalho Chehabbased on groups, or batches of messages, can be beneficial for performance. 254*b9dd2beaSMauro Carvalho Chehab 255*b9dd2beaSMauro Carvalho ChehabOn transmit, there are three ways an application can batch (pipeline) 256*b9dd2beaSMauro Carvalho Chehabmessages on a KCM socket. 257*b9dd2beaSMauro Carvalho Chehab 258*b9dd2beaSMauro Carvalho Chehab 1) Send multiple messages in a single sendmmsg. 259*b9dd2beaSMauro Carvalho Chehab 2) Send a group of messages each with a sendmsg call, where all messages 260*b9dd2beaSMauro Carvalho Chehab except the last have MSG_BATCH in the flags of sendmsg call. 261*b9dd2beaSMauro Carvalho Chehab 3) Create "super message" composed of multiple messages and send this 262*b9dd2beaSMauro Carvalho Chehab with a single sendmsg. 263*b9dd2beaSMauro Carvalho Chehab 264*b9dd2beaSMauro Carvalho ChehabOn receive, the KCM module attempts to queue messages received on the 265*b9dd2beaSMauro Carvalho Chehabsame KCM socket during each TCP ready callback. The targeted KCM socket 266*b9dd2beaSMauro Carvalho Chehabchanges at each receive ready callback on the KCM socket. The application 267*b9dd2beaSMauro Carvalho Chehabdoes not need to configure this. 268*b9dd2beaSMauro Carvalho Chehab 269*b9dd2beaSMauro Carvalho ChehabError handling 270*b9dd2beaSMauro Carvalho Chehab-------------- 271*b9dd2beaSMauro Carvalho Chehab 272*b9dd2beaSMauro Carvalho ChehabAn application should include a thread to monitor errors raised on 273*b9dd2beaSMauro Carvalho Chehabthe TCP connection. Normally, this will be done by placing each 274*b9dd2beaSMauro Carvalho ChehabTCP socket attached to a KCM multiplexor in epoll set for POLLERR 275*b9dd2beaSMauro Carvalho Chehabevent. If an error occurs on an attached TCP socket, KCM sets an EPIPE 276*b9dd2beaSMauro Carvalho Chehabon the socket thus waking up the application thread. When the application 277*b9dd2beaSMauro Carvalho Chehabsees the error (which may just be a disconnect) it should unattach the 278*b9dd2beaSMauro Carvalho Chehabsocket from KCM and then close it. It is assumed that once an error is 279*b9dd2beaSMauro Carvalho Chehabposted on the TCP socket the data stream is unrecoverable (i.e. an error 280*b9dd2beaSMauro Carvalho Chehabmay have occurred in the middle of receiving a message). 281*b9dd2beaSMauro Carvalho Chehab 282*b9dd2beaSMauro Carvalho ChehabTCP connection monitoring 283*b9dd2beaSMauro Carvalho Chehab------------------------- 284*b9dd2beaSMauro Carvalho Chehab 285*b9dd2beaSMauro Carvalho ChehabIn KCM there is no means to correlate a message to the TCP socket that 286*b9dd2beaSMauro Carvalho Chehabwas used to send or receive the message (except in the case there is 287*b9dd2beaSMauro Carvalho Chehabonly one attached TCP socket). However, the application does retain 288*b9dd2beaSMauro Carvalho Chehaban open file descriptor to the socket so it will be able to get statistics 289*b9dd2beaSMauro Carvalho Chehabfrom the socket which can be used in detecting issues (such as high 290*b9dd2beaSMauro Carvalho Chehabretransmissions on the socket). 291