Lines Matching +full:auto +full:- +full:poll
4 - [QUIC Polling API Design](#quic-polling-api-design)
7 …on Past Mistakes in Poller Interface Design](#reflections-on-past-mistakes-in-poller-interface-des…
8 * [Example Use Cases](#example-use-cases)
9 …+ [Use Case A: Simple Blocking or Non-Blocking Application](#use-case-a--simple-blocking-or-non-bl…
10 …+ [Use Case B: Application-Controlled Hierarchical Polling](#use-case-b--application-controlled-hi…
11 * [Use of Poll Descriptors](#use-of-poll-descriptors)
12 * [Event Types and Representation](#event-types-and-representation)
14 + [Sketch A: One-Shot/Immediate Mode API](#sketch-a--one-shot-immediate-mode-api)
15 + [Sketch B: Registered/Retained Mode API](#sketch-b--registered-retained-mode-api)
16 - [Use Case Examples](#use-case-examples)
18 * [Custom Poller Methods](#custom-poller-methods)
20 + [Custom Poller Methods API](#custom-poller-methods-api)
21 + [Internal Polling: Usage within SSL Objects](#internal-polling--usage-within-ssl-objects)
22 + [External Polling: Usage over SSL Objects](#external-polling--usage-over-ssl-objects)
23 …+ [Future Adaptation to Internal Pollable Resources](#future-adaptation-to-internal-pollable-resou…
24 * [Worked Examples](#worked-examples)
25 + [Internal Polling — Default Poll Method](#internal-polling---default-poll-method)
26 + [Internal Polling — Custom Poll Method](#internal-polling---custom-poll-method)
27 + [External Polling — Immediate Mode](#external-polling---immediate-mode)
28 + [External Polling — Retained Mode](#external-polling---retained-mode)
29 …lling — Immediate Mode Without Event Handling](#external-polling---immediate-mode-without-event-ha…
30 * [Change Notification Callback Mechanism](#change-notification-callback-mechanism)
31 * [Q&A](#q-a)
32 * [Windows support](#windows-support)
33 * [Extra features on QUIC objects](#extra-features-on-quic-objects)
34 + [Low-watermark functionality](#low-watermark-functionality)
36 + [Autotick control](#autotick-control)
39 ----------
42 document](quic-server-api.md)), each bound to a single read/write network BIO
43 pair. Therefore an application needs to be able to poll:
45 - a QLSO for new incoming connection events;
46 - a QCSO for new incoming stream events;
47 - a QCSO for new incoming datagram events (when we support the datagram
49 - a QCSO for stream creatability events;
50 - a QCSO for new connection error events;
51 - a QSSO (or QCSO with a default stream attached) for readability events;
52 - a QSSO (or QCSO with a default stream attached) for writeability events;
53 - non-OpenSSL objects, such as OS socket handles.
57 - There are a large number of event types an application might want to poll on.
59 - There are different object types we might want to poll on.
61 - These object types are currently all SSL objects, though we should not assume
64 - The nature of a polling interface means that it must be possible to
65 poll (i.e., block) on all desired objects in a single call. i.e., polling
71 ------------
73 - **Universal cross-pollability.** Ability to poll on any combination of the above
76 - **Support external polling.** An application must be able to be in control
78 or poll(2)-style calls; the application handles all poll(2)-like calls to the
82 - **Support internal polling.** Support a blocking poll(2)-like call provided
85 - **Timeouts.** Support for optional timeouts.
87 - **Multi-threading.** The API must have semantics suitable for performant
88 multi-threaded use, including for concurrent access to the same QUIC objects
94 - Avoid needless impedance discontinuities with COTS polling interfaces (e.g.
95 select(2), poll(2)).
97 - Efficient and performant design.
99 - Future extensibility.
102 -------------------------------------------------------
105 its replacement with poll(2) in POSIX operating systems. To the extent that
106 poll(2) has been replaced, it is largely due to the performance issues it poses
111 The replacements for poll(2) include Linux's epoll(2) and BSD's kqueue(2).
116 - It is designed to poll only FDs; this is probably a partial cause behind
120 - Events registered with epoll are associated with the underlying kernel
126 - There are separate `EPOLL_CTL_ADD` and `EPOLL_CTL_MOD` calls which are needed
131 - Only one FD can be registered, modified, or unregistered per syscall, rather
134 - The design is poorly equipped to handle multithreaded use due to the
142 -----------------
144 Suppose there exists a hypothetical poll(2)-like API called `SSL_poll`. We
147 ### Use Case A: Simple Blocking or Non-Blocking Application
150 be in blocking or non-blocking mode. It wants to block until any of these have
161 ### Use Case B: Application-Controlled Hierarchical Polling
163 An application has two QCSOs open each with one QSSO, all in non-blocking mode.
166 wants to block until various other application-specific non-QUIC events occur.
171 - An application collects file descriptors and event flags to poll from our QUIC
177 - An application does its own polling and timeout handling.
179 - An application calls `SSL_handle_events` if the polling process indicated
180 an event for either of the QUIC poll descriptors or the QUIC event handling
184 - An application calls `SSL_poll` similarly to the above example, but with
196 - An application must be able to use our polling interface without blocking and
199 Use of Poll Descriptors
200 -----------------------
202 As discussed in the [I/O Architecture Design Document](../quic-io-arch.md), the
203 notion of poll descriptors is used to provide an abstraction over arbitrary
207 This concept maps directly to our capacity for application-level polling of the
229 ------------------------------
237 - **R (Readable):** There is application data available to be read.
239 - **W (Writable):** It is currently possible to write more application data.
241 - **ER (Exception on Read):** The receive part of a stream has been remotely
244 - **EW (Exception on Write):** The send part of a stream has been remotely
247 - **EC (Exception on Connection):** A connection has started terminating
250 - **EL (Exception on Listener):** A QUIC listener SSL object has failed,
253 - **ECD (Exception on Connection Drained):** A connection has *finished*
256 - **IC (Incoming Connection):** There is at least one incoming connection
259 - **ISB (Incoming Stream — Bidirectional):** There is at least one
263 - **ISU (Incoming Stream — Unidirectional):** There is at least one
267 - **OSB (Outgoing Stream — Bidirectional):** It is currently possible
270 - **OSU (Outgoing Stream — Unidirectional):** It is currently possible
273 - **F (Failure):** Identifies failure of the `SSL_poll()` mechanism itself.
283 - It is important to provide an adequate granularity of event types so as to
286 - Event types which are not given by a particular object are simply ignored
287 if requested by the application and never raised, similar to `poll(2)`.
289 - While not all event masks may make sense (e.g. `R` but not `ER`), we do not
290 seek to prescribe combinations at this time. This is dissimilar to `poll(2)`
293 - Exception events on some successfully polled resource are not the same as the
303 * -----------------------
347 * ---------------------
401 - There is a use case to learn when we finish terminating because that is when
404 - there is a use case for learning as soon as we start terminating (raised on
407 - shutdown (i.e., waiting for streams to be done transmitting and then
409 application, thus there is no valid need for an application to poll on it.
460 **Q. What happens if this event is enabled on a send-only stream?**
467 A. Potentially on the server side in the future due to incoming 0-RTT data.
492 **Q. What happens if this event is enabled on a send-only stream?**
504 available to be read but this may be a business-as-usual circumstance which the
532 with a stream it can free the QSSO, which means a post-FIN-retirement reset
546 A. No. Applications which wish to know of remotely-triggered send stream part
561 **Q. What happens if this event is enabled on a receive-only stream?**
572 A. Potentially in the future, if 0-RTT is in use and we have a cached 0-RTT
574 more data for 0-RTT.
591 **Q. What happens if this event is enabled on a receive-only stream?**
611 poll for it.
619 protocol-defined semantics, it is important an application can be notified of
657 A. Client — no. Server — no initially, except possibly during 0-RTT when a
679 A. Potentially in future, on the client side only, if 0-RTT is in use and we
680 have a cached 0-RTT session including flow control budgets which establish we
681 have room to write more data for 0-RTT.
698 A. Potentially in future, if 0-RTT is in use; we could receive connection data
716 it is a no-op (similar to `poll(2)` `POLLERR`). This is the only non-maskable
723 non-zero `revents` fields, as always. This includes any `F` events (there may be
724 multiple), and any non-`F` events which were output for earlier entries in the
733 -------
737 - Sketch A: An “immediate-mode” poller interface similar to poll(2).
739 - Sketch B: A “registered” poller interface similar to BSD's kqueue(2) (or Linux's
746 ### Sketch A: One-Shot/Immediate Mode API
757 This structure works similarly to the `struct pollfd` structure used by poll(2).
768 * --------
771 * and determines which poll items have relevant readiness events raised. It is
772 * similar to POSIX poll(2).
784 * Specifying an item with an events field of zero is a no-op; the array entry
785 * is ignored. Unlike poll(2), error events are not automatically included
790 * some kind of object which SSL_poll might (or might not) know how to poll.
791 * Currently, SSL_poll can poll the following kinds of BIO_POLL_DESCRIPTOR:
793 * BIO_POLL_DESCRIPTOR_TYPE_SOCK_FD (int fd) -- OS-pollable sockets only
797 * It cannot be used to poll other OS handle types.
799 * BIO_POLL_DESCRIPTOR_TYPE_SSL (SSL *ssl) -- QUIC SSL objects only
811 * This function can be used in a non-blocking mode where it will provide
813 * even if no item is ready. To facilitate this, pass a zero-value timeout
822 * - SSL_POLL_FLAG_NO_HANDLE_EVENTS:
823 * This may be used only when a zero timeout is specified (non-blocking
828 * will only report pre-existing readiness events for the specified objects.
830 * If timeout is NULL or non-zero, specifying this flag is an error.
834 * the current readiness, or to 0, and *result_count (if non-NULL) is written
846 * is raised on at least one poll item and the function returns 0. At least
858 * This is a "one-shot" API; greater performance may be obtained from using
870 **Performance and thundering-herd issues.** There are two intrinsic performance
873 - Because it does not involve advance registration of things being polled,
878 - Because this design is inherently “stateless”, it cannot really solve
884 This limitation is intrinsic to the design of `poll(2)` and poll-esque APIs.
886 poll(2) and poll(2)-like APIs is widespread and users are likely to appreciate
888 applications which use select/poll, even if those applications suffer impaired
898 - This design can solve the thundering herd problem, achieving efficient
899 distribution of work to threads by auto-disabling an event mask bit after
900 distribution of the readiness event to one thread currently calling the poll
903 - The fundamental call, `SSL_POLL_GROUP_change_poll`, combines the operations
905 important as due to the herd-avoidance design above, events can be and are
906 automatically disarmed and need rearming as frequently as the poll function is
910 - Addition of registered events and mutation of existing events uses an
911 idempotent upsert-type operation, which is what most applications actually
935 * Work queue dispatch (anti-thundering herd) - dispatch to one concurrent call
966 * UINT64_MAX and enable_events to the desired event types. Non-existent
988 * --------------------------
992 * - firstly, if num_changes is non-zero, it updates registered events on the
993 * specified poll group, adding, removing and modifying registered events as
996 * - secondly, if num_events is non-zero, it polls for any events that have
1003 * If num_changes is non-zero, change_stride must be set to
1006 * If num_events is non-zero, event_stride must be set to
1021 * (non-blocking mode); the timeout argument is ignored.
1030 * This function differs from poll-style interfaces in that the events reported
1041 * registered poll descriptor, cookie is set to the cookie value which was
1048 * psuedo-event SSL_POLL_EVENT_POLL_ERROR set, with copies of the desc and
1052 * The 'primary key' for any registered event is the tuple (poll descriptor,
1054 * with the same values for the poll descriptor and instance. The instance field
1056 * poll descriptor. Many applications will be able to use a instance field of
1059 * To unregister an event, pass a matching poll descriptor and instance value
1062 * It is recommended that callers delete a registered event from a poll group
1064 * inside a poll group is freed, the semantics depend on the type of the poll
1066 * socket poll descriptor is closed, therefore it is essential callers
1069 * Other poll descriptor types may implement automatic deregistration from poll
1071 * poll descriptor type. However, even if a poll descriptor type does implement
1074 * may still reference that poll descriptor. Therefore, applications must still
1076 * previously registered in a poll group.
1116 chg->desc = desc;
1117 chg->instance = instance;
1118 chg->cookie = cookie;
1119 chg->disable_events = UINT64_MAX;
1120 chg->enable_events = events;
1121 chg->disable_flags = UINT64_MAX;
1122 chg->enable_flags = flags;
1129 chg->desc = desc;
1130 chg->instance = instance;
1131 chg->cookie.ptr = NULL;
1132 chg->disable_events = 0;
1133 chg->enable_events = 0;
1134 chg->disable_flags = 0;
1135 chg->enable_flags = SSL_POLL_EVENT_FLAG_DELETE;
1145 chg->desc = desc;
1146 chg->instance = instance;
1147 chg->cookie.ptr = NULL;
1148 chg->disable_events = disable_events;
1149 chg->enable_events = enable_events;
1150 chg->disable_flags = 0;
1151 chg->enable_flags = 0;
1161 chg->desc = desc;
1162 chg->instance = instance;
1163 chg->cookie.ptr = NULL;
1164 chg->disable_events = 0;
1165 chg->enable_events = 0;
1166 chg->disable_flags = disable_flags;
1167 chg->enable_flags = enable_flags;
1223 /* Also poll on an ordinary OS socket. */
1248 APP_INFO *app = event->cookie.ptr;
1250 do_something(app, event->revents);
1254 * Scenario 2: Test for pre-existing registered events in non-blocking mode
1279 * Scenario 3: Remove one event but don't poll.
1296 * Scenario 4: Efficient (non-thundering-herd) multi-thread dispatch with
1345 --------
1353 ---------------------
1356 application. This allows an application to support custom poll descriptor types
1357 and provide a way to poll on those poll descriptors. For example, an application
1359 cannot be used with the blocking API) and a custom poller which can poll some
1360 opaque poll descriptor handle provided by the application (which might be e.g.
1369 object. This means it must ultimately make some blocking poll(2)-like call to
1377 - `SSL_poll` will convert the poll descriptors pointing to SSL objects
1378 to network-side poll descriptors by calling `SSL_get_[rw]poll_descriptor`,
1381 - The yielded poll descriptors are then reduced to a set of unique poll
1383 poll descriptor, so duplicates are removed);
1385 - The OS socket handle poll descriptor which was passed in is simply
1386 passed through as-is;
1388 - The resulting set of poll descriptors is then passed on to an underlying
1389 poller implementation, which might be based on e.g. poll(2). But it might
1391 resolved to a custom poll descriptor type.
1393 - When the underlying poll call returns, reverse translation occurs.
1394 Poll descriptors which have become ready in some aspect and which were
1397 poll descriptor). This set of SSL objects is reduced to a unique set of
1402 The above scheme also means that the retained-mode polling API can be more
1404 being re-derived every time.
1410 - Internal polling for blocking API: This is where an SSL object automatically
1412 BIO cannot support a poll descriptor which we understand how to poll on, we
1413 cannot support blocking API operation. We can support a poll descriptor if it
1415 poll it.
1417 - External polling support: This is where an application calls a polling API.
1432 int poll(/* as shown for SSL_poll */);
1455 int (*poll)(SSL_POLL_METHOD *self, /* as shown for SSL_poll */);
1505 if (!--data->refcount)
1515 ++data->refcount;
1528 return d->type == BIO_POLL_DESCRIPTOR_TYPE_SOCK_FD;
1552 data->refcount = 1;
1590 `SSL_CTX`, including if the poll method set on the `SSL_CTX` is changed after
1594 When a poll method is set on a QUIC domain, blocking API calls use that poller
1597 Our QUIC implementation may, if it wishes, use the provided poll method to
1598 construct a poll group, but is not guaranteed to do so. We reserve the right to
1601 later changes the configured poll method by calling `SSL_set1_poll_method`
1604 If the poll method is set to NULL, we use the default poll method, which is the
1607 Because the poll method provided is used to handle blocking on network I/O, a
1608 poll method provided in this context only needs to handle OS socket handles,
1618 Retained-mode usage can be more efficient because it can allow recursive staging
1619 of implementation-specific polling data. For example, suppose an application
1620 enrolls a QCSO and two subsidiary QSSOs in a poll group. The reduction of these
1621 three objects to a single pair of read/write BIO poll descriptors as provided by
1629 to switch to an internal thread-based reactor design, and signal readiness not
1630 via an OS socket handle but via a condition variable or Linux-style `eventfd`.
1634 - For condition variables this would require a new poll descriptor type.
1635 Our default poller could be amended to support this new poll descriptor type.
1640 - For something like `eventfd` things will work well with the existing `SOCK_FD`
1645 opt-in via our API.)
1647 - For something like Win32 Events, `WaitForSingleObject` or
1648 `WaitForMultipleObjects` works, but would require a new poll descriptor type.
1653 ---------------
1655 ### Internal Polling — Default Poll Method
1657 - Application creates a new QCSO
1658 - Application does not set a custom poll method on it
1659 - Application uses it in blocking mode and sets network BIOs
1660 - Our QUIC implementation requests poll descriptors from the network BIOs
1661 - Our QUIC implementation asks the default poller if it understands
1662 how to poll those poll descriptors. If not, blocking cannot be supported.
1663 - When it needs to block, our QUIC implementation uses the default poll method
1664 in either immediate or retained mode based on the poll descriptors reported by
1667 ### Internal Polling — Custom Poll Method
1669 - Application instantiates a custom poll method
1670 - Application creates a new QCSO
1671 - Application sets the custom poll method on the QCSO
1672 - Application configures the QCSO for blocking mode and sets network BIOs
1673 - Our QUIC implementation requests poll descriptors from the network BIOs
1674 - Our QUIC implementation asks the custom poll method if it understands how to
1675 - poll those poll descriptors. If not, blocking cannot be supported.
1676 - When it needs to block, our QUIC implementation uses the custom poll method
1677 in either immediate or retained mode based on the poll descriptors reported
1682 - Application gets a poll method (default or custom)
1683 - Application invokes poll() on the poll method on some number of QLSOs, QCSOs, QSSOs
1685 - The poll method performs translation to a set of OS resources.
1686 - The poll method asks the OS to poll/block.
1687 - The poll method examines the results reported from the OS and performs reverse
1689 - The poll method poll() call reports the results and returns.
1697 - Application gets a poll method (default or custom)
1698 - Application uses the poll method to create a poll group
1699 - Application registers some number of QLSOs, QCSOs, QSSOs and OS sockets, etc.
1700 in the poll group.
1701 - The poll group caches translations to a set of OS resources. It may create
1704 - Application polls using the poll group.
1705 - The poll group asks the OS to poll/block.
1706 - The poll group examines the results reported from the OS and performs reverse
1708 - The poll method reports the results and returns.
1712 - Application gets a poll method (default or custom)
1713 - Application invokes poll() on the poll method on some number of QLSOs, QCSOs,
1715 - If the poll method is the default poll method, it knows how to examine
1717 - If the poll method is a custom poll method, it could choose to subdelegate
1718 this work to the default poll method, or implement it itself.
1721 --------------------------------------
1726 here is that an SSL object registered into a poll group can be automatically
1727 unregistered from that poll group when it is freed.
1766 * For a BIO_CHANGE event, fills the passed pointers if non-NULL with the
1788 ---
1790 **Q. How do we support poll methods which only support immediate mode?**
1795 **Q. How do we support poll methods which only support retained mode?**
1802 (where one poll method's retained mode API maintains state and also makes calls
1803 to another poll method's retained mode API).
1808 uses calls to an OpenSSL external polling API (such as `SSL_poll` or a poll
1811 own, if desired; for example it could create a poll group from the default poll
1812 method and use it to poll only network sockets, some of which may be from QUIC
1813 SSL object poll descriptors, and then if needed call SSL_poll to narrow things
1828 **Q. If an application sets a poll method on both an event leader and a poll
1831 A. Setting a poll method on an event leader provides a mechanism used for internal
1835 what we need to handle if we handle multiple OS-level sockets in future).
1837 Setting a poll method on a poll group provides a mechanism used for polling
1838 using that event group. Note that a custom poll method configured on a SSL
1839 object is **not** used for the translation process performed by a poll group,
1843 **Q. What if different poll methods are configured on different event leaders
1844 (QUIC domains) and an application then tries to poll them all?**
1846 A. Because the poll method configured on an event leader is ignored in favour of
1847 the poll method directly invoked, there is no conflict here. The poll method
1850 **Q. Where should the responsibility for poll descriptor translation lie?**
1852 A. With the poll method or poll group being called at the time.
1856 A. It uses the default poll method. If an application wishes to use a different
1857 poll method, it can call the `poll` method directly on that `BIO_POLL_METHOD`.
1859 **Q. An application creates a poll group, registers an SSL object and later
1860 changes the network BIOs set on that SSL object, or changes the poll descriptors
1865 - An application is not allowed to have the poll descriptors returned by a BIO
1869 - We will need to either:
1871 - have a callback registration interface so retained mode pollers
1872 which have performed cached translation can be notified that a poll
1875 - require retained mode pollers to check for changes to translated objects
1883 **Q. An application creates a poll group, registers a QCSO and some subsidiary
1885 SSL objects auto-deregistered from poller groups?)**
1887 A. We must assume a poll group retains an SSL object pointer if such an object
1890 - require applications to deregister objects from any poll group they are using
1893 - add internal callback registration machinery to QUIC SSL objects so we can
1898 **Q. An application creates a poll group, registers a (non-QUIC-related) OS
1903 application deregisters the handle from the poll group first.
1905 **Q. How does code using a poll method determine what poll descriptors that
1912 ---------------
1917 of the Berkeley select(2)-style API is available, the options for higher
1935 multiple sockets, possibly even multiple poll groups.
1942 4. I/O Completion Ports are the “official” way to do high-performance I/O
1948 which allows a) epoll/kqueue-style interfaces to be built over Winsock, b)
1959 used by third-party software, to the point where it cannot really be changed
1972 option 5 being the preferred approach for projects wanting an epoll/kqueue-style
1980 ------------------------------
1986 ### Low-watermark functionality
1990 is known as “low-watermark” (LOWAT) functionality.
1993 polling-related structures, we propose to add a knob which can be configured on
2006 If `ONESHOT` is set, the low-watermark condition is automatically cleared
2007 after the next call to a read or write function respectively. The low-watermark
2008 condition can also be cleared by passing a low-watermark of 0.
2010 If low-watermark mode is configured, a poller will not report a stream as having
2023 /* All operations - defined as separate bit for forward ABI compatibility */
2076 * If a function is called in non-blocking mode and it cannot execute
2078 * not useful in non-blocking mode, this function can be used to determine if a
2120 Autotick inhibition is only useful in non-blocking mode and it is ignored in
2121 blocking mode. Using it in non-blocking mode carries the following implications:
2123 - Data can be drained using `SSL_read` from existing buffers, but network I/O
2127 - Data can be placed into available write buffer space using `SSL_write`,
2130 - Likewise, no new incoming stream events will occur, and if calls to
2134 - `SSL_do_handshake` will simply report whether the handshake is done or not.