1.. SPDX-License-Identifier: GPL-2.0 2 3============ 4Timestamping 5============ 6 7 81. Control Interfaces 9===================== 10 11The interfaces for receiving network packages timestamps are: 12 13SO_TIMESTAMP 14 Generates a timestamp for each incoming packet in (not necessarily 15 monotonic) system time. Reports the timestamp via recvmsg() in a 16 control message in usec resolution. 17 SO_TIMESTAMP is defined as SO_TIMESTAMP_NEW or SO_TIMESTAMP_OLD 18 based on the architecture type and time_t representation of libc. 19 Control message format is in struct __kernel_old_timeval for 20 SO_TIMESTAMP_OLD and in struct __kernel_sock_timeval for 21 SO_TIMESTAMP_NEW options respectively. 22 23SO_TIMESTAMPNS 24 Same timestamping mechanism as SO_TIMESTAMP, but reports the 25 timestamp as struct timespec in nsec resolution. 26 SO_TIMESTAMPNS is defined as SO_TIMESTAMPNS_NEW or SO_TIMESTAMPNS_OLD 27 based on the architecture type and time_t representation of libc. 28 Control message format is in struct timespec for SO_TIMESTAMPNS_OLD 29 and in struct __kernel_timespec for SO_TIMESTAMPNS_NEW options 30 respectively. 31 32IP_MULTICAST_LOOP + SO_TIMESTAMP[NS] 33 Only for multicast:approximate transmit timestamp obtained by 34 reading the looped packet receive timestamp. 35 36SO_TIMESTAMPING 37 Generates timestamps on reception, transmission or both. Supports 38 multiple timestamp sources, including hardware. Supports generating 39 timestamps for stream sockets. 40 41 421.1 SO_TIMESTAMP (also SO_TIMESTAMP_OLD and SO_TIMESTAMP_NEW) 43------------------------------------------------------------- 44 45This socket option enables timestamping of datagrams on the reception 46path. Because the destination socket, if any, is not known early in 47the network stack, the feature has to be enabled for all packets. The 48same is true for all early receive timestamp options. 49 50For interface details, see `man 7 socket`. 51 52Always use SO_TIMESTAMP_NEW timestamp to always get timestamp in 53struct __kernel_sock_timeval format. 54 55SO_TIMESTAMP_OLD returns incorrect timestamps after the year 2038 56on 32 bit machines. 57 581.2 SO_TIMESTAMPNS (also SO_TIMESTAMPNS_OLD and SO_TIMESTAMPNS_NEW) 59------------------------------------------------------------------- 60 61This option is identical to SO_TIMESTAMP except for the returned data type. 62Its struct timespec allows for higher resolution (ns) timestamps than the 63timeval of SO_TIMESTAMP (ms). 64 65Always use SO_TIMESTAMPNS_NEW timestamp to always get timestamp in 66struct __kernel_timespec format. 67 68SO_TIMESTAMPNS_OLD returns incorrect timestamps after the year 2038 69on 32 bit machines. 70 711.3 SO_TIMESTAMPING (also SO_TIMESTAMPING_OLD and SO_TIMESTAMPING_NEW) 72---------------------------------------------------------------------- 73 74Supports multiple types of timestamp requests. As a result, this 75socket option takes a bitmap of flags, not a boolean. In:: 76 77 err = setsockopt(fd, SOL_SOCKET, SO_TIMESTAMPING, &val, sizeof(val)); 78 79val is an integer with any of the following bits set. Setting other 80bit returns EINVAL and does not change the current state. 81 82The socket option configures timestamp generation for individual 83sk_buffs (1.3.1), timestamp reporting to the socket's error 84queue (1.3.2) and options (1.3.3). Timestamp generation can also 85be enabled for individual sendmsg calls using cmsg (1.3.4). 86 87 881.3.1 Timestamp Generation 89^^^^^^^^^^^^^^^^^^^^^^^^^^ 90 91Some bits are requests to the stack to try to generate timestamps. Any 92combination of them is valid. Changes to these bits apply to newly 93created packets, not to packets already in the stack. As a result, it 94is possible to selectively request timestamps for a subset of packets 95(e.g., for sampling) by embedding an send() call within two setsockopt 96calls, one to enable timestamp generation and one to disable it. 97Timestamps may also be generated for reasons other than being 98requested by a particular socket, such as when receive timestamping is 99enabled system wide, as explained earlier. 100 101SOF_TIMESTAMPING_RX_HARDWARE: 102 Request rx timestamps generated by the network adapter. 103 104SOF_TIMESTAMPING_RX_SOFTWARE: 105 Request rx timestamps when data enters the kernel. These timestamps 106 are generated just after a device driver hands a packet to the 107 kernel receive stack. 108 109SOF_TIMESTAMPING_TX_HARDWARE: 110 Request tx timestamps generated by the network adapter. This flag 111 can be enabled via both socket options and control messages. 112 113SOF_TIMESTAMPING_TX_SOFTWARE: 114 Request tx timestamps when data leaves the kernel. These timestamps 115 are generated in the device driver as close as possible, but always 116 prior to, passing the packet to the network interface. Hence, they 117 require driver support and may not be available for all devices. 118 This flag can be enabled via both socket options and control messages. 119 120SOF_TIMESTAMPING_TX_SCHED: 121 Request tx timestamps prior to entering the packet scheduler. Kernel 122 transmit latency is, if long, often dominated by queuing delay. The 123 difference between this timestamp and one taken at 124 SOF_TIMESTAMPING_TX_SOFTWARE will expose this latency independent 125 of protocol processing. The latency incurred in protocol 126 processing, if any, can be computed by subtracting a userspace 127 timestamp taken immediately before send() from this timestamp. On 128 machines with virtual devices where a transmitted packet travels 129 through multiple devices and, hence, multiple packet schedulers, 130 a timestamp is generated at each layer. This allows for fine 131 grained measurement of queuing delay. This flag can be enabled 132 via both socket options and control messages. 133 134SOF_TIMESTAMPING_TX_ACK: 135 Request tx timestamps when all data in the send buffer has been 136 acknowledged. This only makes sense for reliable protocols. It is 137 currently only implemented for TCP. For that protocol, it may 138 over-report measurement, because the timestamp is generated when all 139 data up to and including the buffer at send() was acknowledged: the 140 cumulative acknowledgment. The mechanism ignores SACK and FACK. 141 This flag can be enabled via both socket options and control messages. 142 143SOF_TIMESTAMPING_TX_COMPLETION: 144 Request tx timestamps on packet tx completion. The completion 145 timestamp is generated by the kernel when it receives packet a 146 completion report from the hardware. Hardware may report multiple 147 packets at once, and completion timestamps reflect the timing of the 148 report and not actual tx time. This flag can be enabled via both 149 socket options and control messages. 150 151 1521.3.2 Timestamp Reporting 153^^^^^^^^^^^^^^^^^^^^^^^^^ 154 155The other three bits control which timestamps will be reported in a 156generated control message. Changes to the bits take immediate 157effect at the timestamp reporting locations in the stack. Timestamps 158are only reported for packets that also have the relevant timestamp 159generation request set. 160 161SOF_TIMESTAMPING_SOFTWARE: 162 Report any software timestamps when available. 163 164SOF_TIMESTAMPING_SYS_HARDWARE: 165 This option is deprecated and ignored. 166 167SOF_TIMESTAMPING_RAW_HARDWARE: 168 Report hardware timestamps as generated by 169 SOF_TIMESTAMPING_TX_HARDWARE or SOF_TIMESTAMPING_RX_HARDWARE 170 when available. 171 172 1731.3.3 Timestamp Options 174^^^^^^^^^^^^^^^^^^^^^^^ 175 176The interface supports the options 177 178SOF_TIMESTAMPING_OPT_ID: 179 Generate a unique identifier along with each packet. A process can 180 have multiple concurrent timestamping requests outstanding. Packets 181 can be reordered in the transmit path, for instance in the packet 182 scheduler. In that case timestamps will be queued onto the error 183 queue out of order from the original send() calls. It is not always 184 possible to uniquely match timestamps to the original send() calls 185 based on timestamp order or payload inspection alone, then. 186 187 This option associates each packet at send() with a unique 188 identifier and returns that along with the timestamp. The identifier 189 is derived from a per-socket u32 counter (that wraps). For datagram 190 sockets, the counter increments with each sent packet. For stream 191 sockets, it increments with every byte. For stream sockets, also set 192 SOF_TIMESTAMPING_OPT_ID_TCP, see the section below. 193 194 The counter starts at zero. It is initialized the first time that 195 the socket option is enabled. It is reset each time the option is 196 enabled after having been disabled. Resetting the counter does not 197 change the identifiers of existing packets in the system. 198 199 This option is implemented only for transmit timestamps. There, the 200 timestamp is always looped along with a struct sock_extended_err. 201 The option modifies field ee_data to pass an id that is unique 202 among all possibly concurrently outstanding timestamp requests for 203 that socket. 204 205 The process can optionally override the default generated ID, by 206 passing a specific ID with control message SCM_TS_OPT_ID (not 207 supported for TCP sockets):: 208 209 struct msghdr *msg; 210 ... 211 cmsg = CMSG_FIRSTHDR(msg); 212 cmsg->cmsg_level = SOL_SOCKET; 213 cmsg->cmsg_type = SCM_TS_OPT_ID; 214 cmsg->cmsg_len = CMSG_LEN(sizeof(__u32)); 215 *((__u32 *) CMSG_DATA(cmsg)) = opt_id; 216 err = sendmsg(fd, msg, 0); 217 218 219SOF_TIMESTAMPING_OPT_ID_TCP: 220 Pass this modifier along with SOF_TIMESTAMPING_OPT_ID for new TCP 221 timestamping applications. SOF_TIMESTAMPING_OPT_ID defines how the 222 counter increments for stream sockets, but its starting point is 223 not entirely trivial. This option fixes that. 224 225 For stream sockets, if SOF_TIMESTAMPING_OPT_ID is set, this should 226 always be set too. On datagram sockets the option has no effect. 227 228 A reasonable expectation is that the counter is reset to zero with 229 the system call, so that a subsequent write() of N bytes generates 230 a timestamp with counter N-1. SOF_TIMESTAMPING_OPT_ID_TCP 231 implements this behavior under all conditions. 232 233 SOF_TIMESTAMPING_OPT_ID without modifier often reports the same, 234 especially when the socket option is set when no data is in 235 transmission. If data is being transmitted, it may be off by the 236 length of the output queue (SIOCOUTQ). 237 238 The difference is due to being based on snd_una versus write_seq. 239 snd_una is the offset in the stream acknowledged by the peer. This 240 depends on factors outside of process control, such as network RTT. 241 write_seq is the last byte written by the process. This offset is 242 not affected by external inputs. 243 244 The difference is subtle and unlikely to be noticed when configured 245 at initial socket creation, when no data is queued or sent. But 246 SOF_TIMESTAMPING_OPT_ID_TCP behavior is more robust regardless of 247 when the socket option is set. 248 249SOF_TIMESTAMPING_OPT_CMSG: 250 Support recv() cmsg for all timestamped packets. Control messages 251 are already supported unconditionally on all packets with receive 252 timestamps and on IPv6 packets with transmit timestamp. This option 253 extends them to IPv4 packets with transmit timestamp. One use case 254 is to correlate packets with their egress device, by enabling socket 255 option IP_PKTINFO simultaneously. 256 257 258SOF_TIMESTAMPING_OPT_TSONLY: 259 Applies to transmit timestamps only. Makes the kernel return the 260 timestamp as a cmsg alongside an empty packet, as opposed to 261 alongside the original packet. This reduces the amount of memory 262 charged to the socket's receive budget (SO_RCVBUF) and delivers 263 the timestamp even if sysctl net.core.tstamp_allow_data is 0. 264 This option disables SOF_TIMESTAMPING_OPT_CMSG. 265 266SOF_TIMESTAMPING_OPT_STATS: 267 Optional stats that are obtained along with the transmit timestamps. 268 It must be used together with SOF_TIMESTAMPING_OPT_TSONLY. When the 269 transmit timestamp is available, the stats are available in a 270 separate control message of type SCM_TIMESTAMPING_OPT_STATS, as a 271 list of TLVs (struct nlattr) of types. These stats allow the 272 application to associate various transport layer stats with 273 the transmit timestamps, such as how long a certain block of 274 data was limited by peer's receiver window. 275 276SOF_TIMESTAMPING_OPT_PKTINFO: 277 Enable the SCM_TIMESTAMPING_PKTINFO control message for incoming 278 packets with hardware timestamps. The message contains struct 279 scm_ts_pktinfo, which supplies the index of the real interface which 280 received the packet and its length at layer 2. A valid (non-zero) 281 interface index will be returned only if CONFIG_NET_RX_BUSY_POLL is 282 enabled and the driver is using NAPI. The struct contains also two 283 other fields, but they are reserved and undefined. 284 285SOF_TIMESTAMPING_OPT_TX_SWHW: 286 Request both hardware and software timestamps for outgoing packets 287 when SOF_TIMESTAMPING_TX_HARDWARE and SOF_TIMESTAMPING_TX_SOFTWARE 288 are enabled at the same time. If both timestamps are generated, 289 two separate messages will be looped to the socket's error queue, 290 each containing just one timestamp. 291 292SOF_TIMESTAMPING_OPT_RX_FILTER: 293 Filter out spurious receive timestamps: report a receive timestamp 294 only if the matching timestamp generation flag is enabled. 295 296 Receive timestamps are generated early in the ingress path, before a 297 packet's destination socket is known. If any socket enables receive 298 timestamps, packets for all socket will receive timestamped packets. 299 Including those that request timestamp reporting with 300 SOF_TIMESTAMPING_SOFTWARE and/or SOF_TIMESTAMPING_RAW_HARDWARE, but 301 do not request receive timestamp generation. This can happen when 302 requesting transmit timestamps only. 303 304 Receiving spurious timestamps is generally benign. A process can 305 ignore the unexpected non-zero value. But it makes behavior subtly 306 dependent on other sockets. This flag isolates the socket for more 307 deterministic behavior. 308 309New applications are encouraged to pass SOF_TIMESTAMPING_OPT_ID to 310disambiguate timestamps and SOF_TIMESTAMPING_OPT_TSONLY to operate 311regardless of the setting of sysctl net.core.tstamp_allow_data. 312 313An exception is when a process needs additional cmsg data, for 314instance SOL_IP/IP_PKTINFO to detect the egress network interface. 315Then pass option SOF_TIMESTAMPING_OPT_CMSG. This option depends on 316having access to the contents of the original packet, so cannot be 317combined with SOF_TIMESTAMPING_OPT_TSONLY. 318 319 3201.3.4. Enabling timestamps via control messages 321^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 322 323In addition to socket options, timestamp generation can be requested 324per write via cmsg, only for SOF_TIMESTAMPING_TX_* (see Section 1.3.1). 325Using this feature, applications can sample timestamps per sendmsg() 326without paying the overhead of enabling and disabling timestamps via 327setsockopt:: 328 329 struct msghdr *msg; 330 ... 331 cmsg = CMSG_FIRSTHDR(msg); 332 cmsg->cmsg_level = SOL_SOCKET; 333 cmsg->cmsg_type = SO_TIMESTAMPING; 334 cmsg->cmsg_len = CMSG_LEN(sizeof(__u32)); 335 *((__u32 *) CMSG_DATA(cmsg)) = SOF_TIMESTAMPING_TX_SCHED | 336 SOF_TIMESTAMPING_TX_SOFTWARE | 337 SOF_TIMESTAMPING_TX_ACK; 338 err = sendmsg(fd, msg, 0); 339 340The SOF_TIMESTAMPING_TX_* flags set via cmsg will override 341the SOF_TIMESTAMPING_TX_* flags set via setsockopt. 342 343Moreover, applications must still enable timestamp reporting via 344setsockopt to receive timestamps:: 345 346 __u32 val = SOF_TIMESTAMPING_SOFTWARE | 347 SOF_TIMESTAMPING_OPT_ID /* or any other flag */; 348 err = setsockopt(fd, SOL_SOCKET, SO_TIMESTAMPING, &val, sizeof(val)); 349 350 3511.4 Bytestream Timestamps 352------------------------- 353 354The SO_TIMESTAMPING interface supports timestamping of bytes in a 355bytestream. Each request is interpreted as a request for when the 356entire contents of the buffer has passed a timestamping point. That 357is, for streams option SOF_TIMESTAMPING_TX_SOFTWARE will record 358when all bytes have reached the device driver, regardless of how 359many packets the data has been converted into. 360 361In general, bytestreams have no natural delimiters and therefore 362correlating a timestamp with data is non-trivial. A range of bytes 363may be split across segments, any segments may be merged (possibly 364coalescing sections of previously segmented buffers associated with 365independent send() calls). Segments can be reordered and the same 366byte range can coexist in multiple segments for protocols that 367implement retransmissions. 368 369It is essential that all timestamps implement the same semantics, 370regardless of these possible transformations, as otherwise they are 371incomparable. Handling "rare" corner cases differently from the 372simple case (a 1:1 mapping from buffer to skb) is insufficient 373because performance debugging often needs to focus on such outliers. 374 375In practice, timestamps can be correlated with segments of a 376bytestream consistently, if both semantics of the timestamp and the 377timing of measurement are chosen correctly. This challenge is no 378different from deciding on a strategy for IP fragmentation. There, the 379definition is that only the first fragment is timestamped. For 380bytestreams, we chose that a timestamp is generated only when all 381bytes have passed a point. SOF_TIMESTAMPING_TX_ACK as defined is easy to 382implement and reason about. An implementation that has to take into 383account SACK would be more complex due to possible transmission holes 384and out of order arrival. 385 386On the host, TCP can also break the simple 1:1 mapping from buffer to 387skbuff as a result of Nagle, cork, autocork, segmentation and GSO. The 388implementation ensures correctness in all cases by tracking the 389individual last byte passed to send(), even if it is no longer the 390last byte after an skbuff extend or merge operation. It stores the 391relevant sequence number in skb_shinfo(skb)->tskey. Because an skbuff 392has only one such field, only one timestamp can be generated. 393 394In rare cases, a timestamp request can be missed if two requests are 395collapsed onto the same skb. A process can detect this situation by 396enabling SOF_TIMESTAMPING_OPT_ID and comparing the byte offset at 397send time with the value returned for each timestamp. It can prevent 398the situation by always flushing the TCP stack in between requests, 399for instance by enabling TCP_NODELAY and disabling TCP_CORK and 400autocork. After linux-4.7, a better way to prevent coalescing is 401to use MSG_EOR flag at sendmsg() time. 402 403These precautions ensure that the timestamp is generated only when all 404bytes have passed a timestamp point, assuming that the network stack 405itself does not reorder the segments. The stack indeed tries to avoid 406reordering. The one exception is under administrator control: it is 407possible to construct a packet scheduler configuration that delays 408segments from the same stream differently. Such a setup would be 409unusual. 410 411 4122 Data Interfaces 413================== 414 415Timestamps are read using the ancillary data feature of recvmsg(). 416See `man 3 cmsg` for details of this interface. The socket manual 417page (`man 7 socket`) describes how timestamps generated with 418SO_TIMESTAMP and SO_TIMESTAMPNS records can be retrieved. 419 420 4212.1 SCM_TIMESTAMPING records 422---------------------------- 423 424These timestamps are returned in a control message with cmsg_level 425SOL_SOCKET, cmsg_type SCM_TIMESTAMPING, and payload of type 426 427For SO_TIMESTAMPING_OLD:: 428 429 struct scm_timestamping { 430 struct timespec ts[3]; 431 }; 432 433For SO_TIMESTAMPING_NEW:: 434 435 struct scm_timestamping64 { 436 struct __kernel_timespec ts[3]; 437 438Always use SO_TIMESTAMPING_NEW timestamp to always get timestamp in 439struct scm_timestamping64 format. 440 441SO_TIMESTAMPING_OLD returns incorrect timestamps after the year 2038 442on 32 bit machines. 443 444The structure can return up to three timestamps. This is a legacy 445feature. At least one field is non-zero at any time. Most timestamps 446are passed in ts[0]. Hardware timestamps are passed in ts[2]. 447 448ts[1] used to hold hardware timestamps converted to system time. 449Instead, expose the hardware clock device on the NIC directly as 450a HW PTP clock source, to allow time conversion in userspace and 451optionally synchronize system time with a userspace PTP stack such 452as linuxptp. For the PTP clock API, see Documentation/driver-api/ptp.rst. 453 454Note that if the SO_TIMESTAMP or SO_TIMESTAMPNS option is enabled 455together with SO_TIMESTAMPING using SOF_TIMESTAMPING_SOFTWARE, a false 456software timestamp will be generated in the recvmsg() call and passed 457in ts[0] when a real software timestamp is missing. This happens also 458on hardware transmit timestamps. 459 4602.1.1 Transmit timestamps with MSG_ERRQUEUE 461^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 462 463For transmit timestamps the outgoing packet is looped back to the 464socket's error queue with the send timestamp(s) attached. A process 465receives the timestamps by calling recvmsg() with flag MSG_ERRQUEUE 466set and with a msg_control buffer sufficiently large to receive the 467relevant metadata structures. The recvmsg call returns the original 468outgoing data packet with two ancillary messages attached. 469 470A message of cm_level SOL_IP(V6) and cm_type IP(V6)_RECVERR 471embeds a struct sock_extended_err. This defines the error type. For 472timestamps, the ee_errno field is ENOMSG. The other ancillary message 473will have cm_level SOL_SOCKET and cm_type SCM_TIMESTAMPING. This 474embeds the struct scm_timestamping. 475 476 4772.1.1.2 Timestamp types 478~~~~~~~~~~~~~~~~~~~~~~~ 479 480The semantics of the three struct timespec are defined by field 481ee_info in the extended error structure. It contains a value of 482type SCM_TSTAMP_* to define the actual timestamp passed in 483scm_timestamping. 484 485The SCM_TSTAMP_* types are 1:1 matches to the SOF_TIMESTAMPING_* 486control fields discussed previously, with one exception. For legacy 487reasons, SCM_TSTAMP_SND is equal to zero and can be set for both 488SOF_TIMESTAMPING_TX_HARDWARE and SOF_TIMESTAMPING_TX_SOFTWARE. It 489is the first if ts[2] is non-zero, the second otherwise, in which 490case the timestamp is stored in ts[0]. 491 492 4932.1.1.3 Fragmentation 494~~~~~~~~~~~~~~~~~~~~~ 495 496Fragmentation of outgoing datagrams is rare, but is possible, e.g., by 497explicitly disabling PMTU discovery. If an outgoing packet is fragmented, 498then only the first fragment is timestamped and returned to the sending 499socket. 500 501 5022.1.1.4 Packet Payload 503~~~~~~~~~~~~~~~~~~~~~~ 504 505The calling application is often not interested in receiving the whole 506packet payload that it passed to the stack originally: the socket 507error queue mechanism is just a method to piggyback the timestamp on. 508In this case, the application can choose to read datagrams with a 509smaller buffer, possibly even of length 0. The payload is truncated 510accordingly. Until the process calls recvmsg() on the error queue, 511however, the full packet is queued, taking up budget from SO_RCVBUF. 512 513 5142.1.1.5 Blocking Read 515~~~~~~~~~~~~~~~~~~~~~ 516 517Reading from the error queue is always a non-blocking operation. To 518block waiting on a timestamp, use poll or select. poll() will return 519POLLERR in pollfd.revents if any data is ready on the error queue. 520There is no need to pass this flag in pollfd.events. This flag is 521ignored on request. See also `man 2 poll`. 522 523 5242.1.2 Receive timestamps 525^^^^^^^^^^^^^^^^^^^^^^^^ 526 527On reception, there is no reason to read from the socket error queue. 528The SCM_TIMESTAMPING ancillary data is sent along with the packet data 529on a normal recvmsg(). Since this is not a socket error, it is not 530accompanied by a message SOL_IP(V6)/IP(V6)_RECVERROR. In this case, 531the meaning of the three fields in struct scm_timestamping is 532implicitly defined. ts[0] holds a software timestamp if set, ts[1] 533is again deprecated and ts[2] holds a hardware timestamp if set. 534 535 5363. Hardware Timestamping configuration: ETHTOOL_MSG_TSCONFIG_SET/GET 537==================================================================== 538 539Hardware time stamping must also be initialized for each device driver 540that is expected to do hardware time stamping. The parameter is defined in 541include/uapi/linux/net_tstamp.h as:: 542 543 struct hwtstamp_config { 544 int flags; /* no flags defined right now, must be zero */ 545 int tx_type; /* HWTSTAMP_TX_* */ 546 int rx_filter; /* HWTSTAMP_FILTER_* */ 547 }; 548 549Desired behavior is passed into the kernel and to a specific device by 550calling the tsconfig netlink socket ``ETHTOOL_MSG_TSCONFIG_SET``. 551The ``ETHTOOL_A_TSCONFIG_TX_TYPES``, ``ETHTOOL_A_TSCONFIG_RX_FILTERS`` and 552``ETHTOOL_A_TSCONFIG_HWTSTAMP_FLAGS`` netlink attributes are then used to set 553the struct hwtstamp_config accordingly. 554 555The ``ETHTOOL_A_TSCONFIG_HWTSTAMP_PROVIDER`` netlink nested attribute is used 556to select the source of the hardware time stamping. It is composed of an index 557for the device source and a qualifier for the type of time stamping. 558 559Drivers are free to use a more permissive configuration than the requested 560configuration. It is expected that drivers should only implement directly the 561most generic mode that can be supported. For example if the hardware can 562support HWTSTAMP_FILTER_PTP_V2_EVENT, then it should generally always upscale 563HWTSTAMP_FILTER_PTP_V2_L2_SYNC, and so forth, as HWTSTAMP_FILTER_PTP_V2_EVENT 564is more generic (and more useful to applications). 565 566A driver which supports hardware time stamping shall update the struct 567with the actual, possibly more permissive configuration. If the 568requested packets cannot be time stamped, then nothing should be 569changed and ERANGE shall be returned (in contrast to EINVAL, which 570indicates that SIOCSHWTSTAMP is not supported at all). 571 572Only a processes with admin rights may change the configuration. User 573space is responsible to ensure that multiple processes don't interfere 574with each other and that the settings are reset. 575 576Any process can read the actual configuration by requesting tsconfig netlink 577socket ``ETHTOOL_MSG_TSCONFIG_GET``. 578 579The legacy configuration is the use of the ioctl(SIOCSHWTSTAMP) with a pointer 580to a struct ifreq whose ifr_data points to a struct hwtstamp_config. 581The tx_type and rx_filter are hints to the driver what it is expected to do. 582If the requested fine-grained filtering for incoming packets is not 583supported, the driver may time stamp more than just the requested types 584of packets. ioctl(SIOCGHWTSTAMP) is used in the same way as the 585ioctl(SIOCSHWTSTAMP). However, this has not been implemented in all drivers. 586 587:: 588 589 /* possible values for hwtstamp_config->tx_type */ 590 enum { 591 /* 592 * no outgoing packet will need hardware time stamping; 593 * should a packet arrive which asks for it, no hardware 594 * time stamping will be done 595 */ 596 HWTSTAMP_TX_OFF, 597 598 /* 599 * enables hardware time stamping for outgoing packets; 600 * the sender of the packet decides which are to be 601 * time stamped by setting SOF_TIMESTAMPING_TX_SOFTWARE 602 * before sending the packet 603 */ 604 HWTSTAMP_TX_ON, 605 }; 606 607 /* possible values for hwtstamp_config->rx_filter */ 608 enum { 609 /* time stamp no incoming packet at all */ 610 HWTSTAMP_FILTER_NONE, 611 612 /* time stamp any incoming packet */ 613 HWTSTAMP_FILTER_ALL, 614 615 /* return value: time stamp all packets requested plus some others */ 616 HWTSTAMP_FILTER_SOME, 617 618 /* PTP v1, UDP, any kind of event packet */ 619 HWTSTAMP_FILTER_PTP_V1_L4_EVENT, 620 621 /* for the complete list of values, please check 622 * the include file include/uapi/linux/net_tstamp.h 623 */ 624 }; 625 6263.1 Hardware Timestamping Implementation: Device Drivers 627-------------------------------------------------------- 628 629A driver which supports hardware time stamping must support the 630ndo_hwtstamp_set NDO or the legacy SIOCSHWTSTAMP ioctl and update the 631supplied struct hwtstamp_config with the actual values as described in 632the section on SIOCSHWTSTAMP. It should also support ndo_hwtstamp_get or 633the legacy SIOCGHWTSTAMP. 634 635Time stamps for received packets must be stored in the skb. To get a pointer 636to the shared time stamp structure of the skb call skb_hwtstamps(). Then 637set the time stamps in the structure:: 638 639 struct skb_shared_hwtstamps { 640 /* hardware time stamp transformed into duration 641 * since arbitrary point in time 642 */ 643 ktime_t hwtstamp; 644 }; 645 646Time stamps for outgoing packets are to be generated as follows: 647 648- In hard_start_xmit(), check if (skb_shinfo(skb)->tx_flags & SKBTX_HW_TSTAMP) 649 is set no-zero. If yes, then the driver is expected to do hardware time 650 stamping. 651- If this is possible for the skb and requested, then declare 652 that the driver is doing the time stamping by setting the flag 653 SKBTX_IN_PROGRESS in skb_shinfo(skb)->tx_flags , e.g. with:: 654 655 skb_shinfo(skb)->tx_flags |= SKBTX_IN_PROGRESS; 656 657 You might want to keep a pointer to the associated skb for the next step 658 and not free the skb. A driver not supporting hardware time stamping doesn't 659 do that. A driver must never touch sk_buff::tstamp! It is used to store 660 software generated time stamps by the network subsystem. 661- Driver should call skb_tx_timestamp() as close to passing sk_buff to hardware 662 as possible. skb_tx_timestamp() provides a software time stamp if requested 663 and hardware timestamping is not possible (SKBTX_IN_PROGRESS not set). 664- As soon as the driver has sent the packet and/or obtained a 665 hardware time stamp for it, it passes the time stamp back by 666 calling skb_tstamp_tx() with the original skb, the raw 667 hardware time stamp. skb_tstamp_tx() clones the original skb and 668 adds the timestamps, therefore the original skb has to be freed now. 669 If obtaining the hardware time stamp somehow fails, then the driver 670 should not fall back to software time stamping. The rationale is that 671 this would occur at a later time in the processing pipeline than other 672 software time stamping and therefore could lead to unexpected deltas 673 between time stamps. 674 6753.2 Special considerations for stacked PTP Hardware Clocks 676---------------------------------------------------------- 677 678There are situations when there may be more than one PHC (PTP Hardware Clock) 679in the data path of a packet. The kernel has no explicit mechanism to allow the 680user to select which PHC to use for timestamping Ethernet frames. Instead, the 681assumption is that the outermost PHC is always the most preferable, and that 682kernel drivers collaborate towards achieving that goal. Currently there are 3 683cases of stacked PHCs, detailed below: 684 6853.2.1 DSA (Distributed Switch Architecture) switches 686^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 687 688These are Ethernet switches which have one of their ports connected to an 689(otherwise completely unaware) host Ethernet interface, and perform the role of 690a port multiplier with optional forwarding acceleration features. Each DSA 691switch port is visible to the user as a standalone (virtual) network interface, 692and its network I/O is performed, under the hood, indirectly through the host 693interface (redirecting to the host port on TX, and intercepting frames on RX). 694 695When a DSA switch is attached to a host port, PTP synchronization has to 696suffer, since the switch's variable queuing delay introduces a path delay 697jitter between the host port and its PTP partner. For this reason, some DSA 698switches include a timestamping clock of their own, and have the ability to 699perform network timestamping on their own MAC, such that path delays only 700measure wire and PHY propagation latencies. Timestamping DSA switches are 701supported in Linux and expose the same ABI as any other network interface (save 702for the fact that the DSA interfaces are in fact virtual in terms of network 703I/O, they do have their own PHC). It is typical, but not mandatory, for all 704interfaces of a DSA switch to share the same PHC. 705 706By design, PTP timestamping with a DSA switch does not need any special 707handling in the driver for the host port it is attached to. However, when the 708host port also supports PTP timestamping, DSA will take care of intercepting 709the ``.ndo_eth_ioctl`` calls towards the host port, and block attempts to enable 710hardware timestamping on it. This is because the SO_TIMESTAMPING API does not 711allow the delivery of multiple hardware timestamps for the same packet, so 712anybody else except for the DSA switch port must be prevented from doing so. 713 714In the generic layer, DSA provides the following infrastructure for PTP 715timestamping: 716 717- ``.port_txtstamp()``: a hook called prior to the transmission of 718 packets with a hardware TX timestamping request from user space. 719 This is required for two-step timestamping, since the hardware 720 timestamp becomes available after the actual MAC transmission, so the 721 driver must be prepared to correlate the timestamp with the original 722 packet so that it can re-enqueue the packet back into the socket's 723 error queue. To save the packet for when the timestamp becomes 724 available, the driver can call ``skb_clone_sk`` , save the clone pointer 725 in skb->cb and enqueue a tx skb queue. Typically, a switch will have a 726 PTP TX timestamp register (or sometimes a FIFO) where the timestamp 727 becomes available. In case of a FIFO, the hardware might store 728 key-value pairs of PTP sequence ID/message type/domain number and the 729 actual timestamp. To perform the correlation correctly between the 730 packets in a queue waiting for timestamping and the actual timestamps, 731 drivers can use a BPF classifier (``ptp_classify_raw``) to identify 732 the PTP transport type, and ``ptp_parse_header`` to interpret the PTP 733 header fields. There may be an IRQ that is raised upon this 734 timestamp's availability, or the driver might have to poll after 735 invoking ``dev_queue_xmit()`` towards the host interface. 736 One-step TX timestamping do not require packet cloning, since there is 737 no follow-up message required by the PTP protocol (because the 738 TX timestamp is embedded into the packet by the MAC), and therefore 739 user space does not expect the packet annotated with the TX timestamp 740 to be re-enqueued into its socket's error queue. 741 742- ``.port_rxtstamp()``: On RX, the BPF classifier is run by DSA to 743 identify PTP event messages (any other packets, including PTP general 744 messages, are not timestamped). The original (and only) timestampable 745 skb is provided to the driver, for it to annotate it with a timestamp, 746 if that is immediately available, or defer to later. On reception, 747 timestamps might either be available in-band (through metadata in the 748 DSA header, or attached in other ways to the packet), or out-of-band 749 (through another RX timestamping FIFO). Deferral on RX is typically 750 necessary when retrieving the timestamp needs a sleepable context. In 751 that case, it is the responsibility of the DSA driver to call 752 ``netif_rx()`` on the freshly timestamped skb. 753 7543.2.2 Ethernet PHYs 755^^^^^^^^^^^^^^^^^^^ 756 757These are devices that typically fulfill a Layer 1 role in the network stack, 758hence they do not have a representation in terms of a network interface as DSA 759switches do. However, PHYs may be able to detect and timestamp PTP packets, for 760performance reasons: timestamps taken as close as possible to the wire have the 761potential to yield a more stable and precise synchronization. 762 763A PHY driver that supports PTP timestamping must create a ``struct 764mii_timestamper`` and add a pointer to it in ``phydev->mii_ts``. The presence 765of this pointer will be checked by the networking stack. 766 767Since PHYs do not have network interface representations, the timestamping and 768ethtool ioctl operations for them need to be mediated by their respective MAC 769driver. Therefore, as opposed to DSA switches, modifications need to be done 770to each individual MAC driver for PHY timestamping support. This entails: 771 772- Checking, in ``.ndo_eth_ioctl``, whether ``phy_has_hwtstamp(netdev->phydev)`` 773 is true or not. If it is, then the MAC driver should not process this request 774 but instead pass it on to the PHY using ``phy_mii_ioctl()``. 775 776- On RX, special intervention may or may not be needed, depending on the 777 function used to deliver skb's up the network stack. In the case of plain 778 ``netif_rx()`` and similar, MAC drivers must check whether 779 ``skb_defer_rx_timestamp(skb)`` is necessary or not - and if it is, don't 780 call ``netif_rx()`` at all. If ``CONFIG_NETWORK_PHY_TIMESTAMPING`` is 781 enabled, and ``skb->dev->phydev->mii_ts`` exists, its ``.rxtstamp()`` hook 782 will be called now, to determine, using logic very similar to DSA, whether 783 deferral for RX timestamping is necessary. Again like DSA, it becomes the 784 responsibility of the PHY driver to send the packet up the stack when the 785 timestamp is available. 786 787 For other skb receive functions, such as ``napi_gro_receive`` and 788 ``netif_receive_skb``, the stack automatically checks whether 789 ``skb_defer_rx_timestamp()`` is necessary, so this check is not needed inside 790 the driver. 791 792- On TX, again, special intervention might or might not be needed. The 793 function that calls the ``mii_ts->txtstamp()`` hook is named 794 ``skb_clone_tx_timestamp()``. This function can either be called directly 795 (case in which explicit MAC driver support is indeed needed), but the 796 function also piggybacks from the ``skb_tx_timestamp()`` call, which many MAC 797 drivers already perform for software timestamping purposes. Therefore, if a 798 MAC supports software timestamping, it does not need to do anything further 799 at this stage. 800 8013.2.3 MII bus snooping devices 802^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 803 804These perform the same role as timestamping Ethernet PHYs, save for the fact 805that they are discrete devices and can therefore be used in conjunction with 806any PHY even if it doesn't support timestamping. In Linux, they are 807discoverable and attachable to a ``struct phy_device`` through Device Tree, and 808for the rest, they use the same mii_ts infrastructure as those. See 809Documentation/devicetree/bindings/ptp/timestamper.txt for more details. 810 8113.2.4 Other caveats for MAC drivers 812^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 813 814Stacked PHCs, especially DSA (but not only) - since that doesn't require any 815modification to MAC drivers, so it is more difficult to ensure correctness of 816all possible code paths - is that they uncover bugs which were impossible to 817trigger before the existence of stacked PTP clocks. One example has to do with 818this line of code, already presented earlier:: 819 820 skb_shinfo(skb)->tx_flags |= SKBTX_IN_PROGRESS; 821 822Any TX timestamping logic, be it a plain MAC driver, a DSA switch driver, a PHY 823driver or a MII bus snooping device driver, should set this flag. 824But a MAC driver that is unaware of PHC stacking might get tripped up by 825somebody other than itself setting this flag, and deliver a duplicate 826timestamp. 827For example, a typical driver design for TX timestamping might be to split the 828transmission part into 2 portions: 829 8301. "TX": checks whether PTP timestamping has been previously enabled through 831 the ``.ndo_eth_ioctl`` ("``priv->hwtstamp_tx_enabled == true``") and the 832 current skb requires a TX timestamp ("``skb_shinfo(skb)->tx_flags & 833 SKBTX_HW_TSTAMP``"). If this is true, it sets the 834 "``skb_shinfo(skb)->tx_flags |= SKBTX_IN_PROGRESS``" flag. Note: as 835 described above, in the case of a stacked PHC system, this condition should 836 never trigger, as this MAC is certainly not the outermost PHC. But this is 837 not where the typical issue is. Transmission proceeds with this packet. 838 8392. "TX confirmation": Transmission has finished. The driver checks whether it 840 is necessary to collect any TX timestamp for it. Here is where the typical 841 issues are: the MAC driver takes a shortcut and only checks whether 842 "``skb_shinfo(skb)->tx_flags & SKBTX_IN_PROGRESS``" was set. With a stacked 843 PHC system, this is incorrect because this MAC driver is not the only entity 844 in the TX data path who could have enabled SKBTX_IN_PROGRESS in the first 845 place. 846 847The correct solution for this problem is for MAC drivers to have a compound 848check in their "TX confirmation" portion, not only for 849"``skb_shinfo(skb)->tx_flags & SKBTX_IN_PROGRESS``", but also for 850"``priv->hwtstamp_tx_enabled == true``". Because the rest of the system ensures 851that PTP timestamping is not enabled for anything other than the outermost PHC, 852this enhanced check will avoid delivering a duplicated TX timestamp to user 853space. 854