Lines Matching +full:controlled +full:- +full:remotely
4 - [QUIC Polling API Design](#quic-polling-api-design)
7 …on Past Mistakes in Poller Interface Design](#reflections-on-past-mistakes-in-poller-interface-des…
8 * [Example Use Cases](#example-use-cases)
9 …+ [Use Case A: Simple Blocking or Non-Blocking Application](#use-case-a--simple-blocking-or-non-bl…
10 …+ [Use Case B: Application-Controlled Hierarchical Polling](#use-case-b--application-controlled-hi…
11 * [Use of Poll Descriptors](#use-of-poll-descriptors)
12 * [Event Types and Representation](#event-types-and-representation)
14 + [Sketch A: One-Shot/Immediate Mode API](#sketch-a--one-shot-immediate-mode-api)
15 + [Sketch B: Registered/Retained Mode API](#sketch-b--registered-retained-mode-api)
16 - [Use Case Examples](#use-case-examples)
18 * [Custom Poller Methods](#custom-poller-methods)
20 + [Custom Poller Methods API](#custom-poller-methods-api)
21 + [Internal Polling: Usage within SSL Objects](#internal-polling--usage-within-ssl-objects)
22 + [External Polling: Usage over SSL Objects](#external-polling--usage-over-ssl-objects)
23 …+ [Future Adaptation to Internal Pollable Resources](#future-adaptation-to-internal-pollable-resou…
24 * [Worked Examples](#worked-examples)
25 + [Internal Polling — Default Poll Method](#internal-polling---default-poll-method)
26 + [Internal Polling — Custom Poll Method](#internal-polling---custom-poll-method)
27 + [External Polling — Immediate Mode](#external-polling---immediate-mode)
28 + [External Polling — Retained Mode](#external-polling---retained-mode)
29 …lling — Immediate Mode Without Event Handling](#external-polling---immediate-mode-without-event-ha…
30 * [Change Notification Callback Mechanism](#change-notification-callback-mechanism)
31 * [Q&A](#q-a)
32 * [Windows support](#windows-support)
33 * [Extra features on QUIC objects](#extra-features-on-quic-objects)
34 + [Low-watermark functionality](#low-watermark-functionality)
36 + [Autotick control](#autotick-control)
39 ----------
42 document](quic-server-api.md)), each bound to a single read/write network BIO
45 - a QLSO for new incoming connection events;
46 - a QCSO for new incoming stream events;
47 - a QCSO for new incoming datagram events (when we support the datagram
49 - a QCSO for stream creatability events;
50 - a QCSO for new connection error events;
51 - a QSSO (or QCSO with a default stream attached) for readability events;
52 - a QSSO (or QCSO with a default stream attached) for writeability events;
53 - non-OpenSSL objects, such as OS socket handles.
57 - There are a large number of event types an application might want to poll on.
59 - There are different object types we might want to poll on.
61 - These object types are currently all SSL objects, though we should not assume
64 - The nature of a polling interface means that it must be possible to
71 ------------
73 - **Universal cross-pollability.** Ability to poll on any combination of the above
76 - **Support external polling.** An application must be able to be in control
78 or poll(2)-style calls; the application handles all poll(2)-like calls to the
82 - **Support internal polling.** Support a blocking poll(2)-like call provided
85 - **Timeouts.** Support for optional timeouts.
87 - **Multi-threading.** The API must have semantics suitable for performant
88 multi-threaded use, including for concurrent access to the same QUIC objects
94 - Avoid needless impedance discontinuities with COTS polling interfaces (e.g.
97 - Efficient and performant design.
99 - Future extensibility.
102 -------------------------------------------------------
116 - It is designed to poll only FDs; this is probably a partial cause behind
120 - Events registered with epoll are associated with the underlying kernel
126 - There are separate `EPOLL_CTL_ADD` and `EPOLL_CTL_MOD` calls which are needed
131 - Only one FD can be registered, modified, or unregistered per syscall, rather
134 - The design is poorly equipped to handle multithreaded use due to the
142 -----------------
144 Suppose there exists a hypothetical poll(2)-like API called `SSL_poll`. We
147 ### Use Case A: Simple Blocking or Non-Blocking Application
150 be in blocking or non-blocking mode. It wants to block until any of these have
161 ### Use Case B: Application-Controlled Hierarchical Polling
163 An application has two QCSOs open each with one QSSO, all in non-blocking mode.
166 wants to block until various other application-specific non-QUIC events occur.
171 - An application collects file descriptors and event flags to poll from our QUIC
177 - An application does its own polling and timeout handling.
179 - An application calls `SSL_handle_events` if the polling process indicated
184 - An application calls `SSL_poll` similarly to the above example, but with
196 - An application must be able to use our polling interface without blocking and
200 -----------------------
202 As discussed in the [I/O Architecture Design Document](../quic-io-arch.md), the
207 This concept maps directly to our capacity for application-level polling of the
229 ------------------------------
237 - **R (Readable):** There is application data available to be read.
239 - **W (Writable):** It is currently possible to write more application data.
241 - **ER (Exception on Read):** The receive part of a stream has been remotely
244 - **EW (Exception on Write):** The send part of a stream has been remotely
247 - **EC (Exception on Connection):** A connection has started terminating
250 - **EL (Exception on Listener):** A QUIC listener SSL object has failed,
253 - **ECD (Exception on Connection Drained):** A connection has *finished*
256 - **IC (Incoming Connection):** There is at least one incoming connection
259 - **ISB (Incoming Stream — Bidirectional):** There is at least one
263 - **ISU (Incoming Stream — Unidirectional):** There is at least one
267 - **OSB (Outgoing Stream — Bidirectional):** It is currently possible
270 - **OSU (Outgoing Stream — Unidirectional):** It is currently possible
273 - **F (Failure):** Identifies failure of the `SSL_poll()` mechanism itself.
283 - It is important to provide an adequate granularity of event types so as to
286 - Event types which are not given by a particular object are simply ignored
289 - While not all event masks may make sense (e.g. `R` but not `ER`), we do not
293 - Exception events on some successfully polled resource are not the same as the
303 * -----------------------
347 * ---------------------
401 - There is a use case to learn when we finish terminating because that is when
404 - there is a use case for learning as soon as we start terminating (raised on
407 - shutdown (i.e., waiting for streams to be done transmitting and then
460 **Q. What happens if this event is enabled on a send-only stream?**
467 A. Potentially on the server side in the future due to incoming 0-RTT data.
492 **Q. What happens if this event is enabled on a send-only stream?**
504 available to be read but this may be a business-as-usual circumstance which the
532 with a stream it can free the QSSO, which means a post-FIN-retirement reset
546 A. No. Applications which wish to know of remotely-triggered send stream part
561 **Q. What happens if this event is enabled on a receive-only stream?**
572 A. Potentially in the future, if 0-RTT is in use and we have a cached 0-RTT
574 more data for 0-RTT.
591 **Q. What happens if this event is enabled on a receive-only stream?**
616 happened on a stream (like it being reset remotely via `STOP_SENDING`) even if
619 protocol-defined semantics, it is important an application can be notified of
657 A. Client — no. Server — no initially, except possibly during 0-RTT when a
679 A. Potentially in future, on the client side only, if 0-RTT is in use and we
680 have a cached 0-RTT session including flow control budgets which establish we
681 have room to write more data for 0-RTT.
698 A. Potentially in future, if 0-RTT is in use; we could receive connection data
716 it is a no-op (similar to `poll(2)` `POLLERR`). This is the only non-maskable
723 non-zero `revents` fields, as always. This includes any `F` events (there may be
724 multiple), and any non-`F` events which were output for earlier entries in the
733 -------
737 - Sketch A: An “immediate-mode” poller interface similar to poll(2).
739 - Sketch B: A “registered” poller interface similar to BSD's kqueue(2) (or Linux's
746 ### Sketch A: One-Shot/Immediate Mode API
768 * --------
784 * Specifying an item with an events field of zero is a no-op; the array entry
793 * BIO_POLL_DESCRIPTOR_TYPE_SOCK_FD (int fd) -- OS-pollable sockets only
799 * BIO_POLL_DESCRIPTOR_TYPE_SSL (SSL *ssl) -- QUIC SSL objects only
811 * This function can be used in a non-blocking mode where it will provide
813 * even if no item is ready. To facilitate this, pass a zero-value timeout
822 * - SSL_POLL_FLAG_NO_HANDLE_EVENTS:
823 * This may be used only when a zero timeout is specified (non-blocking
828 * will only report pre-existing readiness events for the specified objects.
830 * If timeout is NULL or non-zero, specifying this flag is an error.
834 * the current readiness, or to 0, and *result_count (if non-NULL) is written
858 * This is a "one-shot" API; greater performance may be obtained from using
870 **Performance and thundering-herd issues.** There are two intrinsic performance
873 - Because it does not involve advance registration of things being polled,
878 - Because this design is inherently “stateless”, it cannot really solve
884 This limitation is intrinsic to the design of `poll(2)` and poll-esque APIs.
886 poll(2) and poll(2)-like APIs is widespread and users are likely to appreciate
898 - This design can solve the thundering herd problem, achieving efficient
899 distribution of work to threads by auto-disabling an event mask bit after
903 - The fundamental call, `SSL_POLL_GROUP_change_poll`, combines the operations
905 important as due to the herd-avoidance design above, events can be and are
910 - Addition of registered events and mutation of existing events uses an
911 idempotent upsert-type operation, which is what most applications actually
935 * Work queue dispatch (anti-thundering herd) - dispatch to one concurrent call
966 * UINT64_MAX and enable_events to the desired event types. Non-existent
988 * --------------------------
992 * - firstly, if num_changes is non-zero, it updates registered events on the
996 * - secondly, if num_events is non-zero, it polls for any events that have
1003 * If num_changes is non-zero, change_stride must be set to
1006 * If num_events is non-zero, event_stride must be set to
1021 * (non-blocking mode); the timeout argument is ignored.
1030 * This function differs from poll-style interfaces in that the events reported
1048 * psuedo-event SSL_POLL_EVENT_POLL_ERROR set, with copies of the desc and
1116 chg->desc = desc;
1117 chg->instance = instance;
1118 chg->cookie = cookie;
1119 chg->disable_events = UINT64_MAX;
1120 chg->enable_events = events;
1121 chg->disable_flags = UINT64_MAX;
1122 chg->enable_flags = flags;
1129 chg->desc = desc;
1130 chg->instance = instance;
1131 chg->cookie.ptr = NULL;
1132 chg->disable_events = 0;
1133 chg->enable_events = 0;
1134 chg->disable_flags = 0;
1135 chg->enable_flags = SSL_POLL_EVENT_FLAG_DELETE;
1145 chg->desc = desc;
1146 chg->instance = instance;
1147 chg->cookie.ptr = NULL;
1148 chg->disable_events = disable_events;
1149 chg->enable_events = enable_events;
1150 chg->disable_flags = 0;
1151 chg->enable_flags = 0;
1161 chg->desc = desc;
1162 chg->instance = instance;
1163 chg->cookie.ptr = NULL;
1164 chg->disable_events = 0;
1165 chg->enable_events = 0;
1166 chg->disable_flags = disable_flags;
1167 chg->enable_flags = enable_flags;
1248 APP_INFO *app = event->cookie.ptr;
1250 do_something(app, event->revents);
1254 * Scenario 2: Test for pre-existing registered events in non-blocking mode
1296 * Scenario 4: Efficient (non-thundering-herd) multi-thread dispatch with
1345 --------
1353 ---------------------
1369 object. This means it must ultimately make some blocking poll(2)-like call to
1377 - `SSL_poll` will convert the poll descriptors pointing to SSL objects
1378 to network-side poll descriptors by calling `SSL_get_[rw]poll_descriptor`,
1381 - The yielded poll descriptors are then reduced to a set of unique poll
1385 - The OS socket handle poll descriptor which was passed in is simply
1386 passed through as-is;
1388 - The resulting set of poll descriptors is then passed on to an underlying
1393 - When the underlying poll call returns, reverse translation occurs.
1402 The above scheme also means that the retained-mode polling API can be more
1404 being re-derived every time.
1410 - Internal polling for blocking API: This is where an SSL object automatically
1417 - External polling support: This is where an application calls a polling API.
1505 if (!--data->refcount)
1515 ++data->refcount;
1528 return d->type == BIO_POLL_DESCRIPTOR_TYPE_SOCK_FD;
1552 data->refcount = 1;
1618 Retained-mode usage can be more efficient because it can allow recursive staging
1619 of implementation-specific polling data. For example, suppose an application
1629 to switch to an internal thread-based reactor design, and signal readiness not
1630 via an OS socket handle but via a condition variable or Linux-style `eventfd`.
1634 - For condition variables this would require a new poll descriptor type.
1640 - For something like `eventfd` things will work well with the existing `SOCK_FD`
1645 opt-in via our API.)
1647 - For something like Win32 Events, `WaitForSingleObject` or
1653 ---------------
1657 - Application creates a new QCSO
1658 - Application does not set a custom poll method on it
1659 - Application uses it in blocking mode and sets network BIOs
1660 - Our QUIC implementation requests poll descriptors from the network BIOs
1661 - Our QUIC implementation asks the default poller if it understands
1663 - When it needs to block, our QUIC implementation uses the default poll method
1669 - Application instantiates a custom poll method
1670 - Application creates a new QCSO
1671 - Application sets the custom poll method on the QCSO
1672 - Application configures the QCSO for blocking mode and sets network BIOs
1673 - Our QUIC implementation requests poll descriptors from the network BIOs
1674 - Our QUIC implementation asks the custom poll method if it understands how to
1675 - poll those poll descriptors. If not, blocking cannot be supported.
1676 - When it needs to block, our QUIC implementation uses the custom poll method
1682 - Application gets a poll method (default or custom)
1683 - Application invokes poll() on the poll method on some number of QLSOs, QCSOs, QSSOs
1685 - The poll method performs translation to a set of OS resources.
1686 - The poll method asks the OS to poll/block.
1687 - The poll method examines the results reported from the OS and performs reverse
1689 - The poll method poll() call reports the results and returns.
1697 - Application gets a poll method (default or custom)
1698 - Application uses the poll method to create a poll group
1699 - Application registers some number of QLSOs, QCSOs, QSSOs and OS sockets, etc.
1701 - The poll group caches translations to a set of OS resources. It may create
1704 - Application polls using the poll group.
1705 - The poll group asks the OS to poll/block.
1706 - The poll group examines the results reported from the OS and performs reverse
1708 - The poll method reports the results and returns.
1712 - Application gets a poll method (default or custom)
1713 - Application invokes poll() on the poll method on some number of QLSOs, QCSOs,
1715 - If the poll method is the default poll method, it knows how to examine
1717 - If the poll method is a custom poll method, it could choose to subdelegate
1721 --------------------------------------
1766 * For a BIO_CHANGE event, fills the passed pointers if non-NULL with the
1788 ---
1835 what we need to handle if we handle multiple OS-level sockets in future).
1865 - An application is not allowed to have the poll descriptors returned by a BIO
1869 - We will need to either:
1871 - have a callback registration interface so retained mode pollers
1875 - require retained mode pollers to check for changes to translated objects
1885 SSL objects auto-deregistered from poller groups?)**
1890 - require applications to deregister objects from any poll group they are using
1893 - add internal callback registration machinery to QUIC SSL objects so we can
1898 **Q. An application creates a poll group, registers a (non-QUIC-related) OS
1912 ---------------
1917 of the Berkeley select(2)-style API is available, the options for higher
1942 4. I/O Completion Ports are the “official” way to do high-performance I/O
1948 which allows a) epoll/kqueue-style interfaces to be built over Winsock, b)
1959 used by third-party software, to the point where it cannot really be changed
1972 option 5 being the preferred approach for projects wanting an epoll/kqueue-style
1980 ------------------------------
1986 ### Low-watermark functionality
1990 is known as “low-watermark” (LOWAT) functionality.
1993 polling-related structures, we propose to add a knob which can be configured on
2006 If `ONESHOT` is set, the low-watermark condition is automatically cleared
2007 after the next call to a read or write function respectively. The low-watermark
2008 condition can also be cleared by passing a low-watermark of 0.
2010 If low-watermark mode is configured, a poller will not report a stream as having
2023 /* All operations - defined as separate bit for forward ABI compatibility */
2076 * If a function is called in non-blocking mode and it cannot execute
2078 * not useful in non-blocking mode, this function can be used to determine if a
2120 Autotick inhibition is only useful in non-blocking mode and it is ignored in
2121 blocking mode. Using it in non-blocking mode carries the following implications:
2123 - Data can be drained using `SSL_read` from existing buffers, but network I/O
2127 - Data can be placed into available write buffer space using `SSL_write`,
2130 - Likewise, no new incoming stream events will occur, and if calls to
2134 - `SSL_do_handshake` will simply report whether the handshake is done or not.