.\" .\" This file and its contents are supplied under the terms of the .\" Common Development and Distribution License ("CDDL"), version 1.0. .\" You may only use this file in accordance with the terms of version .\" 1.0 of the CDDL. .\" .\" A full copy of the text of the CDDL should have accompanied this .\" source. A copy of the CDDL is also available via the Internet at .\" http://www.illumos.org/license/CDDL. .\" .\" .\" Copyright (c) 2006, Sun Microsystems, Inc. All Rights Reserved. .\" Copyright (c) 2011 Nexenta Systems, Inc. All rights reserved. .\" Copyright 2019 Joyent, Inc. .\" Copyright 2024 Oxide Computer Company .\" Copyright 1989 AT&T .\" .Dd May 2, 2024 .Dt TCP 4P .Os .Sh NAME .Nm tcp , .Nm TCP .Nd Internet Transmission Control Protocol .Sh SYNOPSIS .In sys/socket.h .In netinet/in.h .In netinet/tcp.h .Bd -literal s = socket(AF_INET, SOCK_STREAM, 0); s = socket(AF_INET6, SOCK_STREAM, 0); t = t_open("/dev/tcp", O_RDWR); t = t_open("/dev/tcp6", O_RDWR); .Ed .Sh DESCRIPTION TCP is the virtual circuit protocol of the Internet protocol family. It provides reliable, flow-controlled, in-order, two-way transmission of data. It is a byte-stream protocol layered above the Internet Protocol .Pq Sy IP , or the Internet Protocol Version 6 .Pq Sy IPv6 , the Internet protocol family's internetwork datagram delivery protocol. .Pp Programs can access TCP using the socket interface as a .Dv SOCK_STREAM socket type, or using the Transport Level Interface .Pq Sy TLI where it supports the connection-oriented .Pq Dv BT_COTS_ORD service type. .Pp A checksum over all data helps TCP provide reliable communication. Using a window-based flow control mechanism that makes use of positive acknowledgements, sequence numbers, and a retransmission strategy, TCP can usually recover when datagrams are damaged, delayed, duplicated or delivered out of order by the underlying medium. .Pp TCP provides several socket options, defined in .In netinet/tcp.h and described throughout this document, which may be set using .Xr setsockopt 3SOCKET and read using .Xr getsockopt 3SOCKET . The .Fa level argument for these calls is the protocol number for TCP, available from .Xr getprotobyname 3SOCKET . IP level options may also be used with TCP. See .Xr ip 4P and .Xr ip6 4P . .Ss "Listening And Connecting" TCP uses IP's host-level addressing and adds its own per-host collection of .Dq port addresses . The endpoints of a TCP connection are identified by the combination of an IPv4 or IPv6 address and a TCP port number. Although other protocols, such as the User Datagram Protocol .Pq Sy UDP , may use the same host and port address format, the port space of these protocols is distinct. See .Xr inet 4P and .Xr inet6 4P for details on the common aspects of addressing in the Internet protocol family. .Pp Sockets utilizing TCP are either .Dq active or .Dq passive . Active sockets initiate connections to passive sockets. Passive sockets must have their local IPv4 or IPv6 address and TCP port number bound with the .Xr bind 3SOCKET system call after the socket is created. If an active socket has not been bound by the time .Xr connect 3SOCKET is called, then the operating system will choose a local address and port for the application. By default, TCP sockets are active. A passive socket is created by calling the .Xr listen 3SOCKET system call after binding, which establishes a queueing parameter for the passive socket. Connections to the passive socket can then be received using the .Xr accept 3SOCKET system call. Active sockets use the .Xr connect 3SOCKET call after binding to initiate connections. .Pp If incoming connection requests include an IP source route option, then the reverse source route will be used when responding. .Pp By using the special value .Dv INADDR_ANY with IPv4, or the unspecified address .Pq all zeroes with IPv6, the local IP address can be left unspecified in the .Fn bind call by either active or passive TCP sockets. This feature is usually used if the local address is either unknown or irrelevant. If left unspecified, the local IP address will be bound at connection time to the address of the network interface used to service the connection. For passive sockets, this is the destination address used by the connecting peer. For active sockets, this is usually an address on the same subnet as the destination or default gateway address, although the rules can be more complex. See .Sy "Source Address Selection" in .Xr inet6 4P for a detailed discussion of how this works in IPv6. .Pp Note that no two TCP sockets can be bound to the same port unless the bound IP addresses are different. IPv4 .Dv INADDR_ANY and IPv6 unspecified addresses compare as equal to any IPv4 or IPv6 address. For example, if a socket is bound to .Dv INADDR_ANY or the unspecified address and port .Em N , no other socket can bind to port .Em N , regardless of the binding address. This special consideration of .Dv INADDR_ANY and the unspecified address can be changed using the socket option .Dv SO_REUSEADDR . If .Dv SO_REUSEADDR is set on a socket doing a bind, IPv4 .Dv INADDR_ANY and the IPv6 unspecified address do not compare as equal to any IP address. This means that as long as the two sockets are not both bound to .Dv INADDR_ANY , the unspecified address, or the same IP address, then the two sockets can be bound to the same port. .Pp If an application does not want to allow another socket using the .Dv SO_REUSEADDR option to bind to a port its socket is bound to, the application can set the socket-level .Pq Dv SOL_SOCKET option .Dv SO_EXCLBIND on a socket. The option values of 0 and 1 mean enabling and disabling the option respectively. Once this option is enabled on a socket, no other socket can be bound to the same port. .Ss "Sending And Receiving Data" Once a connection has been established, data can be exchanged using the .Xr read 2 and .Xr write 2 system calls. If, after sending data, the local TCP receives no acknowledgements from its peer for a period of time .Pq for example, if the remote machine crashes , the connection is closed and an error is returned. .Pp When a peer is sending data, it will only send up to the advertised .Dq receive window , which is determined by how much more data the recipient can fit in its buffer. Applications can use the socket-level option .Dv SO_RCVBUF to increase or decrease the receive buffer size. Similarly, the socket-level option .Dv SO_SNDBUF can be used to allow TCP to buffer more unacknowledged and unsent data locally. .Pp Under most circumstances, TCP will send data when it is written by the application. When outstanding data has not yet been acknowledged, though, TCP will gather small amounts of output to be sent as a single packet once an acknowledgement has been received. Usually referred to as Nagle's Algorithm .Pq RFC 896 , this behavior helps prevent flooding the network with many small packets. .Pp However, for some highly interactive clients .Po such as remote shells or windowing systems that send a stream of keypresses or mouse events .Pc , this batching may cause significant delays. To disable this behavior, TCP provides a boolean socket option, .Dv TCP_NODELAY . .Pp Conversely, for other applications, it may be desirable for TCP not to send out any data until a full TCP segment can be sent. To enable this behavior, an application can use the TCP-level socket option .Dv TCP_CORK . When set to a non-zero value, TCP will only send out a full TCP segment. When .Dv TCP_CORK is set to zero after it has been enabled, all currently buffered data is sent out .Po as permitted by the peer's receive window and the current congestion window .Pc . .Pp Still other latency-sensitive applications rely on receiving a quick notification that their packets have been successfully received. To satisfy the requirements of those applications, setting the .Dv TCP_QUICKACK option to a non-zero value will instruct the TCP stack to send an acknowledgment immediately upon receipt of a packet, rather than waiting to acknowledge multiple packets at once. .Pp TCP provides an urgent data mechanism, which may be invoked using the out-of-band provisions of .Xr send 3SOCKET . The caller may mark one byte as .Dq urgent with the .Dv MSG_OOB flag to .Xr send 3SOCKET . This sets an .Dq urgent pointer pointing to this byte in the TCP stream. The receiver on the other side of the stream is notified of the urgent data by a .Dv SIGURG signal. The .Dv SIOCATMARK .Xr ioctl 2 request returns a value indicating whether the stream is at the urgent mark. Because the system never returns data across the urgent mark in a single .Xr read 2 call, it is possible to advance to the urgent data in a simple loop which reads data, testing the socket with the .Dv SIOCATMARK .Fn ioctl request, until it reaches the mark. .Pp The .Dv TCP_MD5SIG option controls the use of MD5 digests .Pq as defined by RFC 2385 on the specified socket. The option value is specified as an .Vt int . When enabled .Pq non-zero , outgoing packets have a digest added to the TCP options in their header, and digests in incoming packets are verified. In order to use this function, TCPSIG security associations .Pq one for each direction must also be configured in the system security association database .Pq SADB using .Xr tcpkey 8 . A listening socket with the option enabled accepts connections with digests only from sources for which a security association exists. Connections without digests are only accepted from sources for which no security association is set up. The resulting connected socket only has TCP_MD5SIG set if the connection is protected with MD5 signatures. If no matching security association .Pq SA is found for traffic on a socket configured with the .Dv TCP_MD5SIG option, no outgoing segments are sent, and all inbound segments are dropped. In particular, the SA must be present prior to the socket being used in a call to .Xr connect 3SOCKET or .Xr accept 3SOCKET . Once the option is enabled and an SA is bound to a connection, it will be cached and used for all subsequent segments; it cannot be changed mid-stream. An SA which is in use can be deleted using .Xr tcpkey 8 and will not be used for any new connections, but existing connections continue to use their cached copy. .Ss "Congestion Control" TCP follows the congestion control algorithm described in RFC 2581, and also supports the initial congestion window .Pq cwnd changes in RFC 3390. The initial cwnd calculation can be overridden by the socket option .Dv TCP_INIT_CWND . An application can use this option to set the initial cwnd to a specified number of TCP segments. This applies to the cases when the connection first starts and restarts after an idle period. The process must have the .Dv PRIV_SYS_NET_CONFIG privilege if it wants to specify a number greater than that calculated by RFC 3390. .Pp The operating system also provides alternative algorithms that may be more appropriate for your application, including the CUBIC congestion control algorithm described in RFC 8312. These can be configured system-wide using .Xr ipadm 8 , or on a per-connection basis with the TCP-level socket option .Dv TCP_CONGESTION , whose argument is the name of the algorithm to use .Pq for example Dq cubic . If the requested algorithm does not exist, then .Fn setsockopt will fail, and .Va errno will be set to .Er ENOENT . .Ss "TCP Keep-Alive" Since TCP determines whether a remote peer is no longer reachable by timing out waiting for acknowledgements, a host that never sends any new data may never notice a peer that has gone away. While consumers can avoid this problem by sending their own periodic heartbeat messages .Pq Transport Layer Security does this, for example, TCP describes an optional keep-alive mechanism in RFC 1122. Applications can enable it using the socket-level option .Dv SO_KEEPALIVE . When enabled, the first keep-alive probe is sent out after a TCP connection is idle for two hours. If the peer does not respond to the probe within eight minutes, the TCP connection is aborted. An application can alter the probe behavior using the following TCP-level socket options: .Bl -tag -offset indent -width 16m .It Dv TCP_KEEPALIVE_THRESHOLD Determines the interval for sending the first probe. The option value is specified as an unsigned integer in milliseconds. The system default is controlled by the TCP .Nm ndd parameter .Cm tcp_keepalive_interval . The minimum value is ten seconds. The maximum is ten days, while the default is two hours. .It Dv TCP_KEEPALIVE_ABORT_THRESHOLD If TCP does not receive a response to the probe, then this option determines how long to wait before aborting a TCP connection. The option value is an unsigned integer in milliseconds. The value zero indicates that TCP should never time out and abort the connection when probing. The system default is controlled by the TCP .Nm ndd parameter .Sy tcp_keepalive_abort_interval . The default is eight minutes. .It Dv TCP_KEEPIDLE This option, like .Dv TCP_KEEPALIVE_THRESHOLD , determines the interval for sending the first probe, except that the option value is an unsigned integer in .Sy seconds . It is provided primarily for compatibility with other Unix flavors. .It Dv TCP_KEEPCNT This option specifies the number of keep-alive probes that should be sent without any response from the peer before aborting the connection. .It Dv TCP_KEEPINTVL This option specifies the interval in seconds between successive, unacknowledged keep-alive probes. .El .Ss "Additional Configuration" illumos supports TCP Extensions for High Performance .Pq RFC 7323 which includes the window scale and timestamp options, and Protection Against Wrap Around Sequence Numbers .Pq Sy PAWS . Note that if timestamps are negotiated on a connection, received segments without timestamps on that connection are silently dropped per the suggestion in the RFC. illumos also supports Selective Acknowledgment .Pq Sy SACK capabilities .Pq RFC 2018 and Explicit Congestion Notification .Pq Sy ECN mechanism .Pq RFC 3168 . .Pp Turn on the window scale option in one of the following ways: .Bl -bullet -offset indent -width 4m .It An application can set .Dv SO_SNDBUF or .Dv SO_RCVBUF size in the .Fn setsockopt option to be larger than 64K. This must be done .Em before the program calls .Fn listen or .Fn connect , because the window scale option is negotiated when the connection is established. Once the connection has been made, it is too late to increase the send or receive window beyond the default TCP limit of 64K. .It For all applications, use .Xr ndd 8 to modify the configuration parameter .Cm tcp_wscale_always . If .Cm tcp_wscale_always is set to .Sy 1 , the window scale option will always be set when connecting to a remote system. If .Cm tcp_wscale_always is .Sy 0 , the window scale option will be set only if the user has requested a send or receive window larger than 64K. The default value of .Cm tcp_wscale_always is .Sy 1 . .It Regardless of the value of .Cm tcp_wscale_always , the window scale option will always be included in a connect acknowledgement if the connecting system has used the option. .El .Pp Turn on SACK capabilities in the following way: .Bl -bullet -offset indent -width 4m .It Use .Nm ndd to modify the configuration parameter .Cm tcp_sack_permitted . If .Cm tcp_sack_permitted is set to .Sy 0 , TCP will not accept SACK or send out SACK information. If .Cm tcp_sack_permitted is set to .Sy 1 , TCP will not initiate a connection with SACK permitted option in the .Sy SYN segment, but will respond with SACK permitted option in the .Sy SYN|ACK segment if an incoming connection request has the SACK permitted option. This means that TCP will only accept SACK information if the other side of the connection also accepts SACK information. If .Cm tcp_sack_permitted is set to .Sy 2 , it will both initiate and accept connections with SACK information. The default for .Cm tcp_sack_permitted is .Sy 2 .Pq active enabled . .El .Pp Turn on the TCP ECN mechanism in the following way: .Bl -bullet -offset indent -width 4m .It Use .Nm ndd to modify the configuration parameter .Cm tcp_ecn_permitted . If .Cm tcp_ecn_permitted is set to .Sy 0 , then TCP will not negotiate with a peer that supports ECN mechanism. If .Cm tcp_ecn_permitted is set to .Sy 1 when initiating a connection, TCP will not tell a peer that it supports .Sy ECN mechanism. However, it will tell a peer that it supports .Sy ECN mechanism when accepting a new incoming connection request if the peer indicates that it supports .Sy ECN mechanism in the .Sy SYN segment. If .Cm tcp_ecn_permitted is set to 2, in addition to negotiating with a peer on .Sy ECN mechanism when accepting connections, TCP will indicate in the outgoing .Sy SYN segment that it supports .Sy ECN mechanism when TCP makes active outgoing connections. The default for .Cm tcp_ecn_permitted is 1. .El .Pp Turn on the timestamp option in the following way: .Bl -bullet -offset indent -width 4m .It Use .Nm ndd to modify the configuration parameter .Cm tcp_tstamp_always . If .Cm tcp_tstamp_always is .Sy 1 , the timestamp option will always be set when connecting to a remote machine. If .Cm tcp_tstamp_always is .Sy 0 , the timestamp option will not be set when connecting to a remote system. The default for .Cm tcp_tstamp_always is .Sy 0 . .It Regardless of the value of .Cm tcp_tstamp_always , the timestamp option will always be included in a connect acknowledgement .Pq and all succeeding packets if the connecting system has used the timestamp option. .El .Pp Use the following procedure to turn on the timestamp option only when the window scale option is in effect: .Bl -bullet -offset indent -width 4m .It Use .Nm ndd to modify the configuration parameter .Cm tcp_tstamp_if_wscale . Setting .Cm tcp_tstamp_if_wscale to .Sy 1 will cause the timestamp option to be set when connecting to a remote system, if the window scale option has been set. If .Cm tcp_tstamp_if_wscale is .Sy 0 , the timestamp option will not be set when connecting to a remote system. The default for .Cm tcp_tstamp_if_wscale is .Sy 1 . .El .Pp Protection Against Wrap Around Sequence Numbers .Pq Sy PAWS is always used when the timestamp option is set. .Pp The operating system also supports multiple methods of generating initial sequence numbers. One of these methods is the improved technique suggested in RFC 1948. We .Em HIGHLY recommend that you set sequence number generation parameters as close to boot time as possible. This prevents sequence number problems on connections that use the same connection-ID as ones that used a different sequence number generation. The .Sy svc:/network/initial:default service configures the initial sequence number generation. The service reads the value contained in the configuration file .Pa /etc/default/inetinit to determine which method to use. .Pp The .Pa /etc/default/inetinit file is an unstable interface, and may change in future releases. .Sh EXAMPLES .Ss Example 1: Connecting to a server .Bd -literal $ gcc -std=c99 -Wall -lsocket -o client client.c $ cat client.c #include #include #include #include #include #include #include int main(int argc, char *argv[]) { struct addrinfo hints, *gair, *p; int fd, rv, rlen; char buf[1024]; int y = 1; if (argc != 3) { fprintf(stderr, "%s \en", argv[0]); return (1); } memset(&hints, 0, sizeof (hints)); hints.ai_family = PF_UNSPEC; hints.ai_socktype = SOCK_STREAM; if ((rv = getaddrinfo(argv[1], argv[2], &hints, &gair)) != 0) { fprintf(stderr, "getaddrinfo() failed: %s\en", gai_strerror(rv)); return (1); } for (p = gair; p != NULL; p = p->ai_next) { if ((fd = socket( p->ai_family, p->ai_socktype, p->ai_protocol)) == -1) { perror("socket() failed"); continue; } if (connect(fd, p->ai_addr, p->ai_addrlen) == -1) { close(fd); perror("connect() failed"); continue; } break; } if (p == NULL) { fprintf(stderr, "failed to connect to server\en"); return (1); } freeaddrinfo(gair); if (setsockopt(fd, SOL_SOCKET, SO_KEEPALIVE, &y, sizeof (y)) == -1) { perror("setsockopt(SO_KEEPALIVE) failed"); return (1); } while ((rlen = read(fd, buf, sizeof (buf))) > 0) { fwrite(buf, rlen, 1, stdout); } if (rlen == -1) { perror("read() failed"); } fflush(stdout); if (close(fd) == -1) { perror("close() failed"); } return (0); } $ ./client 127.0.0.1 8080 hello $ ./client ::1 8080 hello .Ed .Ss Example 2: Accepting client connections .Bd -literal $ gcc -std=c99 -Wall -lsocket -o server server.c $ cat server.c #include #include #include #include #include #include #include #include void logmsg(struct sockaddr *s, int bytes) { char dq[INET6_ADDRSTRLEN]; switch (s->sa_family) { case AF_INET: { struct sockaddr_in *s4 = (struct sockaddr_in *)s; inet_ntop(AF_INET, &s4->sin_addr, dq, sizeof (dq)); fprintf(stdout, "sent %d bytes to %s:%d\en", bytes, dq, ntohs(s4->sin_port)); break; } case AF_INET6: { struct sockaddr_in6 *s6 = (struct sockaddr_in6 *)s; inet_ntop(AF_INET6, &s6->sin6_addr, dq, sizeof (dq)); fprintf(stdout, "sent %d bytes to [%s]:%d\en", bytes, dq, ntohs(s6->sin6_port)); break; } default: fprintf(stdout, "sent %d bytes to unknown client\en", bytes); break; } } int main(int argc, char *argv[]) { struct addrinfo hints, *gair, *p; int sfd, cfd; int slen, wlen, rv; if (argc != 3) { fprintf(stderr, "%s \en", argv[0]); return (1); } slen = strlen(argv[2]); memset(&hints, 0, sizeof (hints)); hints.ai_family = PF_UNSPEC; hints.ai_socktype = SOCK_STREAM; hints.ai_flags = AI_PASSIVE; if ((rv = getaddrinfo(NULL, argv[1], &hints, &gair)) != 0) { fprintf(stderr, "getaddrinfo() failed: %s\en", gai_strerror(rv)); return (1); } for (p = gair; p != NULL; p = p->ai_next) { if ((sfd = socket( p->ai_family, p->ai_socktype, p->ai_protocol)) == -1) { perror("socket() failed"); continue; } if (bind(sfd, p->ai_addr, p->ai_addrlen) == -1) { close(sfd); perror("bind() failed"); continue; } break; } if (p == NULL) { fprintf(stderr, "server failed to bind()\en"); return (1); } freeaddrinfo(gair); if (listen(sfd, 1024) != 0) { perror("listen() failed"); return (1); } fprintf(stdout, "waiting for clients...\en"); for (int times = 0; times < 5; times++) { struct sockaddr_storage stor; socklen_t alen = sizeof (stor); struct sockaddr *addr = (struct sockaddr *)&stor; if ((cfd = accept(sfd, addr, &alen)) == -1) { perror("accept() failed"); continue; } wlen = 0; do { wlen += write(cfd, argv[2] + wlen, slen - wlen); } while (wlen < slen); logmsg(addr, wlen); if (close(cfd) == -1) { perror("close(cfd) failed"); } } if (close(sfd) == -1) { perror("close(sfd) failed"); } fprintf(stdout, "finished.\en"); return (0); } $ ./server 8080 $'hello\en' waiting for clients... sent 6 bytes to [::ffff:127.0.0.1]:59059 sent 6 bytes to [::ffff:127.0.0.1]:47448 sent 6 bytes to [::ffff:127.0.0.1]:54949 sent 6 bytes to [::ffff:127.0.0.1]:55186 sent 6 bytes to [::1]:62256 finished. .Ed .Sh DIAGNOSTICS A socket operation may fail if: .Bl -tag -offset indent -width 16m .It Er EISCONN A .Fn connect operation was attempted on a socket on which a .Fn connect operation had already been performed. .It Er ETIMEDOUT A connection was dropped due to excessive retransmissions. .It Er ECONNRESET The remote peer forced the connection to be closed (usually because the remote machine has lost state information about the connection due to a crash). .It Er ECONNREFUSED The remote peer actively refused connection establishment (usually because no process is listening to the port). .It Er EADDRINUSE A .Fn bind operation was attempted on a socket with a network address/port pair that has already been bound to another socket. .It Er EADDRNOTAVAIL A .Fn bind operation was attempted on a socket with a network address for which no network interface exists. .It Er EACCES A .Fn bind operation was attempted with a .Dq reserved port number and the effective user ID of the process was not the privileged user. .It Er ENOBUFS The system ran out of memory for internal data structures. .El .Sh SEE ALSO .Xr svcs 1 , .Xr ioctl 2 , .Xr read 2 , .Xr write 2 , .Xr accept 3SOCKET , .Xr bind 3SOCKET , .Xr connect 3SOCKET , .Xr getprotobyname 3SOCKET , .Xr getsockopt 3SOCKET , .Xr listen 3SOCKET , .Xr send 3SOCKET , .Xr inet 4P , .Xr inet6 4P , .Xr ip 4P , .Xr ip6 4P , .Xr smf 7 , .Xr ndd 8 , .Xr svcadm 8 , .Xr tcpkey 8 .Rs .%A "K. Ramakrishnan" .%A "S. Floyd" .%A "D. Black" .%T "The Addition of Explicit Congestion Notification (ECN) to IP" .%R "RFC 3168" .%D "September 2001" .Re .Rs .%A "M. Mathias" .%A "J. Mahdavi" .%A "S. Ford" .%A "A. Romanow" .%T "TCP Selective Acknowledgement Options" .%R "RFC 2018" .%D "October 1996" .Re .Rs .%A "S. Bellovin" .%T "Defending Against Sequence Number Attacks" .%R "RFC 1948" .%D "May 1996" .Re .Rs .%A "D. Borman" .%A "B. Braden" .%A "V. Jacobson" .%A "R. Scheffenegger, Ed." .%T "TCP Extensions for High Performance" .%R "RFC 7323" .%D "September 2014" .Re .Rs .%A "Jon Postel" .%T "Transmission Control Protocol - DARPA Internet Program Protocol Specification" .%R "RFC 793" .%C "Network Information Center, SRI International, Menlo Park, CA." .%D "September 1981" .Re .Rs .%A "A. Heffernan" .%T "Protection of BGP Sessions via the TCP MD5 Signature Option" .%R "RFC 2385" .%D "August 1998" .Re .Sh NOTES The .Sy tcp service is managed by the service management facility, .Xr smf 7 , under the service identifier .Sy svc:/network/initial:default . .Pp Administrative actions on this service, such as enabling, disabling, or requesting restart, can be performed using .Xr svcadm 8 . The service's status can be queried using the .Xr svcs 1 command.