quic-design/server/quic-server-api.md

QUIC Server API Design
======================

Requirements
------------

Essential:

- Support server mode.
- Support multiple clients on one UDP socket/BIO.
- Accept incoming clients and get a `SSL *` object which works just like a
  client QCSO.

Maybe:

- Binding multiple BIOs (UDP sockets) to one listener?

Implied:

- Support connection migration and preferred addressing in the future.
- Maximise reuse of existing APIs.

Approach
--------

Two design sketches are shown below, their relative merits are evaluated, and
one is chosen.

Sketch A: `SSL_LISTENER`
------------------------

A new type `SSL_LISTENER` is introduced to the public API. This represents an
entity which listens for incoming connections. It is **not** intended to be QUIC
specific and provides an abstract interface which could be adapted to other
cases in the future. For example, it could be used for TLS over TCP to make
accepting incoming TCP connections easy.

Internally, an `SSL_LISTENER` has a common header in much the same way that
`SSL` objects now do. For example, if called using a QUIC `SSL_CTX`, the
internal structure might be a `QUIC_LISTENER`, etc. If called using a TLS
`SSL_CTX`, this might be a `TCP_LISTENER` doing something very ordinary. We can
support DTLS in the future in the same way as QUIC, etc.

Internally, an `SSL_LISTENER` stores any common state that needs to be held
collectively for all connections created by that listener.

`SSL_LISTENER` objects are refcounted and referenced by any connection which
descends from them. Thus, an application may free the `SSL_LISTENER`, but
it will be retained until it is no longer depended on by any connection.

The operation of an `SSL_LISTENER` is simple: call
`SSL_LISTENER_accept_connection`. For QUIC, this returns a new QCSO, which is
considered a child of the listener.

```c
SSL_LISTENER *SSL_LISTENER_new(SSL_CTX *ctx, BIO *net_rbio, BIO *net_wbio);
int SSL_LISTENER_up_ref(SSL_LISTENER *l);
int SSL_LISTENER_free(SSL_LISTENER *l);

BIO *SSL_LISTENER_get0_net_rbio(SSL_LISTENER *l);
BIO *SSL_LISTENER_get0_net_wbio(SSL_LISTENER *l);
int SSL_LISTENER_set1_net_rbio(SSL_LISTENER *l, BIO *net_rbio);
int SSL_LISTENER_set1_net_wbio(SSL_LISTENER *l, BIO *net_wbio);

SSL *SSL_LISTENER_accept_connection(SSL_LISTENER *l, uint64_t flags);
size_t SSL_LISTENER_get_accept_queue_len(SSL_LISTENER *l);
```

### Discussion

However, there are a number of disadvantages to introducing a new object type.
The concept of a listener shares many of the same concepts inherent to
supporting non-blocking network I/O which our SSL API for QUIC connections has
been carefully designed to support:

- Poll descriptors (`SSL_get_[rw]poll_descriptor`);
- `SSL_net_(read|write)_desired`;
- `SSL_handle_events`;
- `SSL_get_event_timeout`;
- BIO configuration (`SSL_set_[rw]bio`);
- Refcounting (`SSL_up_ref`, `SSL_free`).

We would need to duplicate all of these APIs. Moreover application developers
would need to use both and the distinction between the two would become
confusing.

Sketch B: New SSL Object Type: Listener
---------------------------------------

Instead of the above, the definition of a new SSL object subtype — a Listener —
is proposed. A QLSO (QUIC Listener SSL Object) is defined in much the same way
that QCSOs and QSSOs are currently defined, using the common SSL structure
header.

Many existing APIs around the handling of non-blocking I/O can then be supported
and work in exactly the same way.

```c
/*
 * SSL_new_listener
 * ----------------
 *
 * Creates a QLSO, assuming a QUIC SSL_CTX.
 *
 * TCP and DTLS can be supported in the future; not supported for non-QUIC
 * SSL_CTXs for now.
 */
SSL *SSL_new_listener(SSL_CTX *ctx, uint64_t flags);

/*
 * A QLSO supports the following existing APIs:
 *
 *   - Poll descriptors (`SSL_get_[rw]poll_descriptor`)
 *   - `SSL_net_(read|write)_desired`
 *   - `SSL_handle_events`
 *   - `SSL_get_event_timeout`
 *   - BIO configuration (`SSL_set_[rw]bio`)
 *   - Refcounting (`SSL_up_ref`, `SSL_free`)
 */

/*
 * New API for QLSOs: SSL_accept_connection
 * ----------------------------------------
 *
 * These APIs are exactly the same as  SSL_accept_stream, but made a separate
 * function to avoid confusion. Supported on QLSOs only.
 */
SSL *SSL_accept_connection(SSL *ssl, uint64_t flags);
size_t SSL_get_accept_connection_queue_len(SSL *ssl);

/*
 * New API for QLSOs: SSL_listen
 * -----------------------------
 *
 * This is not usually needed as SSL_accept_connection will implicitly trigger
 * this if it has not been called yet. It is only needed if a server wishes to
 * ensure it has started to accept incoming connections but does not wish to
 * actually call SSL_accept_connection yet. In this regard it is similar to
 * listen(2).
 */
int SSL_listen(SSL *ssl);

/*
 * New API for QCSOs: SSL_get_peer_addr
 * ------------------------------------
 *
 * An API to retrieve a L4 peer address. Supported on QCSOs only.
 */
int SSL_get_peer_addr(SSL *ssl, BIO_ADDR *peer_addr);

/* SSL_get_local_addr may be introduced once it is relevant (migration etc.) */

/* Returns 1 for QLSOs (or future TLS/DTLS listeners) only. */
int SSL_is_listener(SSL *ssl);

/* Returns the QLSO (if any) from which a QCSO or QSSO is descended. */
SSL *SSL_get0_listener(SSL *ssl);

/* SSL_is_quic returns 1 */
```

Selection of Design
-------------------

We could also consider using an `SSL_CTX` as the listener type. Since a
`SSL_CTX` is intended to span the lifespan of multiple TLS connections there is
a degree of logic here. However, this does not match up naturally because an
`SSL_CTX` is not designed to hold network resources or manage network I/O. It
does not have the facilities for polling and event handling that the `SSL`
object API has been enhanced with. It would also be a fairly major paradigm
shift for existing server applications and make things difficult if a server
wanted to use different `SSL_CTX` instances to use different e.g. TLS settings
within the same listener.

Based on the merits of the above, Sketch B is chosen for adoption.

Discussion of Relevant Issues
-----------------------------

### Notes on Internal Refactoring to Support Multiple Connections

Currently a client connection is a QCSO
(`QUIC_CONNECTION`) which contains a `QUIC_CHANNEL`. The `QUIC_CONNECTION`
provides API Personality Layer (APL) implementation and wraps the APL-agnostic
`QUIC_CHANNEL`. We currently have two APLs, the libssl API (“the” APL) and
`QUIC_TSERVER` for internal testing use. `QUIC_XSO` is the APL object
representing a QUIC stream and wraps our APL-agnostic channel-owned
`QUIC_STREAM` object. There is no specific object in the `QUIC_TSERVER` APL for
a stream as they are referenced in that API by stream ID only.

`QUIC_CHANNEL` corresponds exactly to a QUIC connection. In a similar manner,
`QUIC_STREAM` corresponds exactly to a QUIC stream. In server operation,
multiple `QUIC_CHANNEL` instances will exist, one per accepted connection.

Above we introduce the notion of a QUIC Listener SSL Object (QLSO), which
internally will be represented by the type `QUIC_LISTENER`. This forms a new
object in the libssl APL.

With server operation, multiple connections will share a single UDP
socket/network BIO. Certain resources currently tied 1:1 into a `QUIC_CHANNEL`
will be broken out, such as the network BIO pointers and `QUIC_DEMUX` instance.
A `QUIC_CHANNEL` continues to own e.g. the QRX/QTX as these are
connection-specific.

Because preserving the separation of the APL(s) and the QUIC core is desirable,
the concept of a `QUIC_PORT` is introduced. This is the internal object tracking
a demux and network BIOs which a `QUIC_CHANNEL` will belong to. In other words,
the new situation is as follows:

| Abbr. | libssl APL Object | QUIC Core Object |
|-------|-------------------|------------------|
| QLSO  | `QUIC_LISTENER`   | `QUIC_PORT`      |
| QCSO  | `QUIC_CONNECTION` | `QUIC_CHANNEL`   |
| QSSO  | `QUIC_XSO`        | `QUIC_STREAM`    |

### Listener Flexibility

A listener allows multiple connections to be received on a UDP socket. However,
there is no intrinsic requirement in QUIC that a socket only be used for client
use, or only for server use. Therefore, it should be possible for an application
to create a QLSO, use it to accept connections, and also use it to initiate
outgoing connections.

This can be supported in a simple manner:

```c
SSL *SSL_new_from_listener(SSL *ssl);
```

This creates an unconnected (idle) QCSO from a QLSO. Use of
`SSL_set1_initial_peer_addr` is mandatory in this case as peer address
autodetection is inhibited.

The created QCSO otherwise works in a completely normal fashion.

### Client-Only Listeners

In fact, the realisation of this hybrid capability is simply an emergent
consequence of our internal refactor. Since a QUIC server will be split into an
internal `QUIC_PORT` and zero or more `QUIC_CONNECTION` instances, so too will a
client connection become simply a degenerate case where there is a `QUIC_PORT`
and only one `QUIC_CONNECTION`. So the client case basically just becomes a
simplified API where we spin up a `QUIC_PORT`/listener automatically to support
a single connection.

It is potentially desirable for a client to be able to use the listener
API even if it is only initiating connections as a client and never accepting
connections. There are several reasons why an application might want to do this:

- it allows the application to centralise its I/O polling and networking arrangements;
- it allows the application to handle the TERMINATING phase of connections in a
  better (and more RFC-compliant) way.

For the latter reason, consider how connection termination works. Once a QUIC
Immediate Close is initiated using a `CONNECTION_CLOSE` frame, a connection
enters the `TERMINATING` state and is expected to stay there for some period of
time until the possibility of various contingencies (such as loss of a
`CONNECTION_CLOSE` frame and its necessary retransmission) has been accounted
for. However, many applications may not wish to hang around for potentially
several seconds or longer while tearing down a QUIC connection.

Allowing clients to use a listener resolves this problem, if the lifetime of the
application exceeds the lifetime of most client connections. The listener can
continue to hold the `QUIC_PORT` internally and respond to enquiries for
terminating and terminated connections, even if (as far as the application is
concerned) no client connections currently exist and all prior client
connections have been freed using `SSL_free`. The internal `QUIC_CHANNEL` can be
kept around by the `QUIC_PORT` until it reaches the `TERMINATED` state.

A client-only listener can be created one of two ways:

1. by calling `SSL_new_listener` while passing this flag:

    ```
    #define SSL_LISTENER_FLAG_NO_ACCEPT  (1U << 0)
    ```

2. by using a client `SSL_METHOD`, which implies `NO_ACCEPT`.

This flag simply indicates that this listener is not to be used to accept
incoming connections. A client can then use `SSL_new_from_listener` as usual to
create an outgoing QCSO.

### CID Length Determination

Currently a client connection uses a zero-length local CID. Since we want to
support multiple outgoing client connections on one socket these will need to
support a non-zero-length local CID. Equally servers will need to support CIDs.

By default when a QLSO is used, we will use a real, non-zero-length CID. If the
existing QUIC client API is used (with no QLSO) we will likely spin up a
`QUIC_PORT` internally but specially choose to use a zero-length local CID since
we know this port will never be used for more than one connection.

If an application creates a QLSO explicitly but knows it will not need to
create more than one connection, it can pass the following flag when creating
the listener:

```c
#define SSL_LISTENER_FLAG_SINGLE_CONN   (1U << 1)
```

Failure to do so is nonfatal and simply increases packet overheads.

### Future Compatibility: TLS over TCP Support

Assuming we choose to use the SSL object type as the basis of a new listener
object type, let us consider how this API could also be used to support TLS over
TCP, at least in a server role:

```c
/* Blocking example: */
{
    SSL *l, *conn;
    BIO *tcp_listener = get_bio_sock_for_listening_tcp_socket();
    SSL_CTX *ctx = create_ssl_ctx(TLS_server_method());
    BIO_ADDR addr;

    l = SSL_new_listener(ctx);
    SSL_set_bio(l, tcp_listener, tcp_listener);

    for (;;) {
        /* standard SSL_CONNECTION is returned */
        conn = SSL_accept_connection(l, 0);

        spawn_thread_to_process_conn(conn, &addr);
    }
}
```

Essentially, when `SSL_new_listener` is given a `SSL_CTX` which is using a TLS
method, it returns some different kind of object, say a `TCP_LISTENER`, which
also is a kind of SSL object. `SSL_accept_connection` calls `accept(2)` via some
BIO abstraction on the provided BIO, and constructs a new BIO on the resulting
socket, wrapping it via standard `SSL_new` using the same `SSL_CTX`.

Thus, this API should be compatible with any future adaptation to also support
TCP.

### Future Compatibility: DTLS over UDP Support

The relevant concerns regarding DTLS are likely to be highly similar to QUIC,
so this API should also be adaptable to support DTLS in the future, providing
a unified server-side API. `SSL_new_listener` will return e.g. a
`DTLS_LISTENER`.

### Supporting Multiple Sockets

There are three means that we can provide to support a server which wants to
listen on multiple UDP sockets:

1. Allow a QLSO to bind multiple BIOs;
2. Create multiple QLSOs and use some polling mechanism to service all of them;
3. Create some higher level object which can aggregate multiple QLSOs.

(1) would require more extensive API changes and should only be done if a
compelling reason to do so arises. (3) is a more advanced design which is
theoretically possible in future and is discussed in the next section.
For now, we will support only (2) (which we can support via our planned polling
API) until a compelling need arises.

### Roles and Domains: Concepts and Future Evolution

A client can use QUIC using a single QCSO without ever creating a QLSO.
Conversely a client or server can use QUIC by creating a QLSO and then one or
more subsidiary QCSOs. This implies that, at least in terms of the publicly
visible API, the roles of certain essential aspects of QUIC can belong to
different publicly visible API objects.

The following concepts are introduced:

- **Port Leader:** A Port Leader is the APL object which caused the creation
  of a `QUIC_PORT`, which is responsible for servicing a particular pair
  of network BIOs. Currently maps to one UDP socket.

  In the simple client API, the QCSO is the port leader.
  In a client using a QLSO, the QLSO is the port leader.
  In a server, the QLSO is the port leader.

- **Event Leader:** An Event Leader is the APL object which handles event
  processing. Calls to SSL_handle_events() for any subsidiary object is
  equivalent to a call to SSL_handle_events() on the Event Leader, which handles
  state machine processing for that object and all subsidiary objects.

  The event leader is the highest-level APL object in a QUIC APL object
  hierarchy:

  In the simple client API, the QCSO is the event leader.
  In a client using a QLSO, the QLSO is the event leader.
  In a server, the QLSO is the event leader.

- **Domain:** A QUIC domain is a set of APL objects rooted in an Event Leader.
  In other words, requests to handle events on subsidiary objects in a domain
  are channeled up to the Event Leader and processed there. A QUIC domain is a
  set of logically grouped QUIC objects which share a single I/O reactor and
  event processing context.

  A domain could contain multiple QLSOs (i.e., multiple UDP sockets) in future.
  In this case QLSOs may cease to be the most senior APL object type and there
  may be some new hypothetical SSL_new_quic_domain() call.

  However, there is no plan to implement domains at this time; for the time
  being, this is purely conceptual and ensures our architecture can grow to
  supporting multiple sockets and listeners in a given QUIC context if need be.

  If ever implemented, a domain will be reified by a `QUIC_DOMAIN` structure as
  part of the APL and a `QUIC_ENGINE` structure as part of the QUIC core. The
  SSL object type will be abbreviated QDSO (QUIC Domain SSL Object).

The introduction of the concept of a domain demonstrates the value of the Port
Leader and Domain Leader concepts going forward:

Example 1 — Explicitly Created Domain:

```text
QDSO (QUIC_DOMAIN/QUIC_ENGINE)              Event Leader
    QLSO (QUIC_LISTENER/QUIC_PORT)          Port Leader
        QCSO (QUIC_CONNECTION/QUIC_CHANNEL)
            QSSO (QUIC_XSO/QUIC_STREAM)
```

Example 2 - Explicitly Created QLSO, Implicitly Created Domain:

```text
QLSO (QUIC_LISTENER/QUIC_PORT)              Event Leader, Port Leader
    [QUIC_ENGINE]
    QCSO (QUIC_CONNECTION/QUIC_CHANNEL)
        QSSO (QUIC_XSO/QUIC_STREAM)
```

Here we see that the QLSO now has the Event Leader role, as it is the most
senior object visible publicly as a constructed API object. Note that a
QUIC_ENGINE will be created internally to allow the QLSO to service the Event
Leader role, but the corresponding QUIC_DOMAIN APL object will not exist, as
this is not API-visible.

Example 3 - Explicitly Created Single Client QCSO:

```text
QCSO (QUIC_CONNECTION/QUIC_CHANNEL)         Event Leader, Port Leader
    [QUIC_ENGINE]
    [QUIC_PORT]
    QSSO (QUIC_XSO/QUIC_STREAM)
```

Here we see that the QCSO now has the Port Leader role. Note that a QUIC_PORT
will be created internally to allow the QCSO to service the Port Leader role,
but the corresponding QUIC_LISTENER APL object will not exist, as this is not
API-visible. Similarly a QUIC_ENGINE is spun up to handle the Event Leader role,
as above.

It should be emphasised that everything in this section is largely academic and
intended to demonstrate potential future directions and the adaptability of the
design to future evolution. There is no immediate plan to implement a QDSO
object type in the APL. However it will be viable to do so if we want to support
multiple sockets per domain someday.

| Abbr. | libssl APL Object | QUIC Core Object | Seniority   | Potential Roles |
|-------|-------------------|------------------|-------------|-----------------|
| QDSO  | `QUIC_DOMAIN`     | `QUIC_ENGINE`    | 1 (Highest) | EL              |
| QLSO  | `QUIC_LISTENER`   | `QUIC_PORT`      | 2           | EL PL           |
| QCSO  | `QUIC_CONNECTION` | `QUIC_CHANNEL`   | 3           | EL PL           |
| QSSO  | `QUIC_XSO`        | `QUIC_STREAM`    | 4           |                 |

### Polling

We will support polling all QUIC object types (QLSOs, QCSOs, QSSOs) using a new
polling API. This is the subject of its own design and will be discussed in more
detail in a separate design document.

### Impact of `SSL_METHOD` selection

The interaction of our various QCSO and QLSO construction APIs in the APL,
and the choice of the `SSL_METHOD` for a `SSL_CTX`, must also be considered.
The following interactions are chosen:

```text
SSL_new(QUIC_client_method)             → QCSO(Client)
SSL_new(QUIC_client_thread_method)      → QCSO(Client)
SSL_new(QUIC_server_method)             → (not supported; error)

SSL_new_listener(QUIC_client_method)        → QLSO(Client)   (implicit NO_ACCEPT)
SSL_new_listener(QUIC_client_thread_method) → QLSO(Client)   (implicit NO_ACCEPT)
SSL_new_listener(QUIC_server_method)        → QLSO(Server + Client)

SSL_new_from_listener(QUIC_client_method)           → QCSO (Client)
SSL_new_from_listener(QUIC_client_thread_method)    → QCSO (Client)
SSL_new_from_listener(QUIC_server_method)           → QCSO (Client)
```

Observations:

- It makes no sense to try and make a single QCSO proactively as a server,
  so this just results in an error.

- A server QLSO can also make outgoing client connections.

- By contrast, if a client `SSL_METHOD` is specified when creating a listener,
  it is assumed to be uninterested in accepting connections and this implies the
  `NO_ACCEPT` flag. In this regard the functionality of a server is a superset
  of the functionality of a client.

- `SSL_new_from_listener` always creates a client QCSO.

- `SINGLE_CONN` is implied automatically only when calling `SSL_new` and not
  using a QLSO.

Usage Example
-------------

The following is a blocking example, where `SSL_accept_connection` waits until a
new connection is available.

```c
{
    SSL *l, *conn;
    SSL_CTX *ctx = create_ssl_ctx(QUIC_server_method());
    BIO *dgram_bio = get_dgram_bio();

    l = SSL_new_listener(ctx);
    SSL_set_bio(l, dgram_bio, dgram_bio);

    for (;;) {
        /* automatically starts and calls SSL_listen on first call */
        conn = SSL_accept_connection(l, 0); /* standard QCSO is returned */

        spawn_thread_to_process_conn(conn);
    }
}
```

In non-blocking mode, `SSL_accept_connection` will simply act in a non-blocking
fashion and return immediately with NULL if no new incoming connection is
available, just like `SSL_accept_stream`. The details of how polling and waiting
for incoming stream events, and other events, will work will be discussed in the
subsequent polling design document.