1*7f2fe78bSCy Schubert<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> 2*7f2fe78bSCy Schubert<html xmlns="https://www.w3.org/1999/xhtml"> 3*7f2fe78bSCy Schubert<head> 4*7f2fe78bSCy Schubert<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" /> 5*7f2fe78bSCy Schubert<title>Windows CCAPI RPC design</title> 6*7f2fe78bSCy Schubert<style type="text/css"> 7*7f2fe78bSCy Schubert<!-- 8*7f2fe78bSCy Schubert.style2 {color: 0} 9*7f2fe78bSCy Schubert.style3 {font-family: "Courier New", Courier, monospace} 10*7f2fe78bSCy Schubert.style5 {color: #CC3300} 11*7f2fe78bSCy Schubert.style6 {color: #999999} 12*7f2fe78bSCy Schubert.style7 {color: #000099} 13*7f2fe78bSCy Schubert--> 14*7f2fe78bSCy Schubert</style> 15*7f2fe78bSCy Schubert</head> 16*7f2fe78bSCy Schubert 17*7f2fe78bSCy Schubert<body> 18*7f2fe78bSCy Schubert<h2 class="style5">Proposed RPC design for Windows CCAPI clients and server</h2> 19*7f2fe78bSCy Schubert<p>The proposal is for a single user; the solution is replicated for each user logged onto the PC.</p> 20*7f2fe78bSCy Schubert<h2>Conventions & clarifications </h2> 21*7f2fe78bSCy Schubert<p>"Client" and "server" refer to the CCAPI client and server. </p> 22*7f2fe78bSCy Schubert<p>The CCAPI client acts as both an RPC client and RPC server and the CCAPI server acts as both an RPC client and RPC server. </p> 23*7f2fe78bSCy Schubert<ul> 24*7f2fe78bSCy Schubert <li class="style2">The RPC call from the CCAPI client to the CCAPI server is called the "request." In this mode, the CCAPI client is the RPC client and the CCAPI server is the RPC server.</li> 25*7f2fe78bSCy Schubert <li class="style2">The RPC call from the CCAPI server to the CCAPI client is called the "reply." In this mode, the CCAPI client is the RPC server and the CCAPI server is the RPC client. </li> 26*7f2fe78bSCy Schubert</ul> 27*7f2fe78bSCy Schubert<p>The Windows username is referred to below as "<USER>." </p> 28*7f2fe78bSCy Schubert<p>The Windows Logon Security Identifier is referred to as "<LSID>." </p> 29*7f2fe78bSCy Schubert<p><UUID> means a thread-specific UUID.</p> 30*7f2fe78bSCy Schubert<p><SST> means <strong>s</strong>erver <strong>s</strong>tart <strong>t</strong>ime, a time_t. </p> 31*7f2fe78bSCy Schubert<p>A description of client and server authentication has not been added yet.</p> 32*7f2fe78bSCy Schubert<h2>Design Requirements </h2> 33*7f2fe78bSCy Schubert<ul> 34*7f2fe78bSCy Schubert <li>The server's OS-independent code is single threaded, because it must operate on platforms that do not allow multiple threads. </li> 35*7f2fe78bSCy Schubert <li>The client and server must be able to maintain connections, where state is maintained between individual messages.</li> 36*7f2fe78bSCy Schubert <li>Individual messages must be handled in a single threaded server. </li> 37*7f2fe78bSCy Schubert <li>The server must be able to detect when a client dies, so that any connection state can be cleaned up. </li> 38*7f2fe78bSCy Schubert</ul> 39*7f2fe78bSCy Schubert<h2>Design</h2> 40*7f2fe78bSCy Schubert<p>The server and each client create an RPC endpoint. The server's endpoint is CCS_<LSID> and the client's endpoint is CCAPI_<UUID>, where each client geta a UUID. </p> 41*7f2fe78bSCy Schubert<p>On Windows, the server's ccs_pipe_t type is a char* and is set to the client UUID.</p> 42*7f2fe78bSCy Schubert<h3>How is the request handled in the server and the reply sent to the client? </h3> 43*7f2fe78bSCy Schubert<p>One straightforward way is for the reply to be the returned data in the request RPC call (an [out] parameter). That is, data passed from the RPC server to the RPC client. The request handler calls <span class="style3">ccs_server_handle_request</span>. Eventually, the server code calls <span class="style3">ccs_os_server_send_reply,</span> which saves the reply somewhere. When the server eventually returns to the request handler, the handler returns the saved reply to the client.</p> 44*7f2fe78bSCy Schubert<p>But this doesn't work. If two clients A and B ask for the same lock, A will acquire the lock and B will have to wait. But if the single threaded server waits for B's lock, it will never handle A's unlock message. Therefore the server must return to B's request handler and <em>not</em> send a reply to B. So this method will not work. </p> 45*7f2fe78bSCy Schubert<p>Instead, there are listener and worker threads in Windows-specific code. </p> 46*7f2fe78bSCy Schubert<p>The client's <span class="style3">cci_os_ipc </span>function waits for <span class="style3">ccs_reply</span>. The client sends the request, including <em>its UUID,</em> from which the server can construct the endpoint on which to call <span class="style3">ccs_reply</span>. </p> 47*7f2fe78bSCy Schubert<p>The server's listener thread listens for RPC requests. The request handler puts each request/<em>reply</em> endpoint in a queue and returns to the client.</p> 48*7f2fe78bSCy Schubert<p>The server's worker thread removes items from the queue, calls <span class="style3">ccs_server_handle_request</span>. <span class="style3">ccs_server_handle_request</span> takes both the request data and the client UUID . Eventually <span class="style3">ccs_os_server_send_reply</span> is called, with the reply data and client UUID in the reply_pipe. <span class="style3">ccs_os_server_send_reply</span> calls <span class="style3">ccs_reply</span> on the client's endpoint, which sends the reply to the client. </p> 49*7f2fe78bSCy Schubert<p>Is there any security issue with the client listening for RPC calls from the server?</p> 50*7f2fe78bSCy Schubert<h3>Connections</h3> 51*7f2fe78bSCy Schubert<p>If the client wants state to be maintained on the server, the client creates a connection. When the connection is closed, the server cleans up any state associated with the connection. </p> 52*7f2fe78bSCy Schubert<p>Any given thread in an application process could want to create a connection. When cci_ipc_thread_init is called, the connection thread-local variables are initialized. New connections are created when cci_os_ipc() (via _cci_ipc_send) is called and no connection was previously established. Basically we lazily establish connections so the client doesn't talk to the server until it has to.</p> 53*7f2fe78bSCy Schubert<h3>Detecting client exit</h3> 54*7f2fe78bSCy Schubert<p>The server must be able to detect when clients disappear, so the server can free any resources that had been held for the client. </p> 55*7f2fe78bSCy Schubert<p>The Windows RPC API does not appear to provide a notification for an endpoint disappearing. It does provide a way to ask if an endpoint is listening. This is useful for polling, but we want a better performing solution than that. </p> 56*7f2fe78bSCy Schubert<p>The client has an <em>isAlive</em> function on its endpoint. </p> 57*7f2fe78bSCy Schubert<p>To detect the client disappearing without using polling, the server makes an asynchronous call to the <em>isAlive</em> function on the client's endpoint. The <em>isAlive</em> function never returns. When the client exits for any reason, its <em>endpoint</em> will be closed and the server's function call will return an error. The asynchronous call on the server means no additional threads are used. </p> 58*7f2fe78bSCy Schubert<p>Windows provides a number of notification methods to signal I/O completion. Among them are I/O completion ports and callback functions. I chose callback functions because they appear to consume fewer resources. </p> 59*7f2fe78bSCy Schubert<h3>RPC Endpoint / Function summary</h3> 60*7f2fe78bSCy Schubert<ul> 61*7f2fe78bSCy Schubert <li>The server creates one CCS_<LSID> endpoint to listen for connection requests and client requests. 62*7f2fe78bSCy Schubert It has the functions 63*7f2fe78bSCy Schubert <ul> 64*7f2fe78bSCy Schubert <li>ccs_rpc_connect(msgtype, UUIDlen, <UUID>, status)</li> 65*7f2fe78bSCy Schubert <li>ccs_rpc_request(msgtype, UUIDlen, <UUID>, msglen, msg, SST, status) called by client. NB: The windows server sets the <span class="style3">in_client_pipe </span>to the <span class="style3">in_reply_pipe</span>.<br /> 66*7f2fe78bSCy Schubert </li> 67*7f2fe78bSCy Schubert </ul> 68*7f2fe78bSCy Schubert </li> 69*7f2fe78bSCy Schubert <li>Each client thread creates a CCAPI_<UUID> endpoint. It has the functions 70*7f2fe78bSCy Schubert <ul> 71*7f2fe78bSCy Schubert <li>isAlive [function never returns.] </li> 72*7f2fe78bSCy Schubert <li>ccs_rpc_request_reply(msgtype, SST, replylen, reply, status)</li> 73*7f2fe78bSCy Schubert <li>ccs_rpc_connect_reply(msgtype, SST, status</li> 74*7f2fe78bSCy Schubert </ul> 75*7f2fe78bSCy Schubert </li> 76*7f2fe78bSCy Schubert</ul> 77*7f2fe78bSCy Schubert<h2>Windows-specific implementation details </h2> 78*7f2fe78bSCy Schubert<h3>Client CCAPI library initialization:</h3> 79*7f2fe78bSCy Schubert<p>This code runs when the CCAPI DLL is loaded. </p> 80*7f2fe78bSCy Schubert<ul> 81*7f2fe78bSCy Schubert <li>?</li> 82*7f2fe78bSCy Schubert</ul> 83*7f2fe78bSCy Schubert<h3>Client initialization: </h3> 84*7f2fe78bSCy Schubert<p>This code runs when cci_os_ipc_thread_init is called: </p> 85*7f2fe78bSCy Schubert<ul> 86*7f2fe78bSCy Schubert <li>Generate <UUID> and save in thread-specific storage. This serves as the client ID / ccs_pipe_t. </li> 87*7f2fe78bSCy Schubert <li>Create client endpoint.</li> 88*7f2fe78bSCy Schubert <li>Listen on client <em></em>endpoint.</li> 89*7f2fe78bSCy Schubert <li>Create canonical server connection endpoint from the <LSID>, which the client and server should have in common. </li> 90*7f2fe78bSCy Schubert <li>Test if server is listening to the CCS_<LSID> endpoint. 91*7f2fe78bSCy Schubert <ul> 92*7f2fe78bSCy Schubert <li>If not, quit. (! Start it?) </li> 93*7f2fe78bSCy Schubert </ul> 94*7f2fe78bSCy Schubert </li> 95*7f2fe78bSCy Schubert <li>Call ccs_connect(<UUID>) on the CCS_<LSID> endpoint.</li> 96*7f2fe78bSCy Schubert <li>Save SST in thread-specific storage. </li> 97*7f2fe78bSCy Schubert</ul> 98*7f2fe78bSCy Schubert<h3>Server initialization:</h3> 99*7f2fe78bSCy Schubert<p class="style6">[old]</p> 100*7f2fe78bSCy Schubert<ul> 101*7f2fe78bSCy Schubert <li class="style6">Server is initialized by client starting a new process. There should be only one server process per Windows username. </li> 102*7f2fe78bSCy Schubert</ul> 103*7f2fe78bSCy Schubert<p class="style7">[new]</p> 104*7f2fe78bSCy Schubert<ul> 105*7f2fe78bSCy Schubert <li class="style7">Server is started by kfwlogon (as is done currently). </li> 106*7f2fe78bSCy Schubert <li>Capture <strong>s</strong>erver <strong>s</strong>tart <strong>t</strong>ime (SST). </li> 107*7f2fe78bSCy Schubert <li>Start listener thread, create listener endpoint, listen on CCS_<LSID> endpoint. </li> 108*7f2fe78bSCy Schubert</ul> 109*7f2fe78bSCy Schubert<h3>Establishing a connection: </h3> 110*7f2fe78bSCy Schubert<ul> 111*7f2fe78bSCy Schubert <li>Client calls ccs_connect(<UUID>) on server's CCS_<LSID> endpoint.</li> 112*7f2fe78bSCy Schubert <li>Client gets back and stores SST in thread-specific storage.</li> 113*7f2fe78bSCy Schubert <li>If new connection, server ... 114*7f2fe78bSCy Schubert <ul> 115*7f2fe78bSCy Schubert <li>adds connection to connection table</li> 116*7f2fe78bSCy Schubert <li>calls isAlive on CCAPI_<UUID>. 117*7f2fe78bSCy Schubert <ul> 118*7f2fe78bSCy Schubert <li>NB: isAlive never returns. </li> 119*7f2fe78bSCy Schubert </ul> 120*7f2fe78bSCy Schubert </li> 121*7f2fe78bSCy Schubert </ul> 122*7f2fe78bSCy Schubert </li> 123*7f2fe78bSCy Schubert</ul> 124*7f2fe78bSCy Schubert<h3>Client request:</h3> 125*7f2fe78bSCy Schubert<p>The server's reply to the client's request is not synchronous.</p> 126*7f2fe78bSCy Schubert<ul> 127*7f2fe78bSCy Schubert <li>Client calls ccs_rpc_request(msglen, msg, msgtype, UUIDlen, <UUID>, SST, status) on server's endpoint.</li> 128*7f2fe78bSCy Schubert <li>Server listen thread receives message, queues request.</li> 129*7f2fe78bSCy Schubert <li>Server worker thread dequeues request, processes, calls ccs_rpc_reply(replylen, reply, msgtype, status) on CCAPI_<UUID>.</li> 130*7f2fe78bSCy Schubert <li>Server checks SST. If server's SST is different, it means server has restarted since client created connection. </li> 131*7f2fe78bSCy Schubert <li>Client receives reply. </li> 132*7f2fe78bSCy Schubert</ul> 133*7f2fe78bSCy Schubert<h3>Detecting client exit</h3> 134*7f2fe78bSCy Schubert<ul> 135*7f2fe78bSCy Schubert <li>When connection created, client created an endpoint.</li> 136*7f2fe78bSCy Schubert <li>Server calls isAlive on client's endpoint. </li> 137*7f2fe78bSCy Schubert <li>When isAlive returns, the server's notification callback will be called. Call back routine queues a DISCONNECT pseudo-message. When the server's worker thread handles the DISCONNECT, it will release connection resources.</li> 138*7f2fe78bSCy Schubert</ul> 139*7f2fe78bSCy Schubert<h3>Detecting server exit </h3> 140*7f2fe78bSCy Schubert<ul> 141*7f2fe78bSCy Schubert <li>Client's call to ccs_rpc_request will return an error if the server has gone away. </li> 142*7f2fe78bSCy Schubert</ul> 143*7f2fe78bSCy Schubert<p> </p> 144*7f2fe78bSCy Schubert<p>------<br /> 145*7f2fe78bSCy Schubert Stop: <br /> 146*7f2fe78bSCy SchubertStart: </p> 147*7f2fe78bSCy Schubert</body> 148*7f2fe78bSCy Schubert</html> 149