11da177e4SLinus Torvalds /* 21da177e4SLinus Torvalds * INET An implementation of the TCP/IP protocol suite for the LINUX 31da177e4SLinus Torvalds * operating system. INET is implemented using the BSD Socket 41da177e4SLinus Torvalds * interface as the means of communication with the user level. 51da177e4SLinus Torvalds * 61da177e4SLinus Torvalds * Implementation of the Transmission Control Protocol(TCP). 71da177e4SLinus Torvalds * 802c30a84SJesper Juhl * Authors: Ross Biro 91da177e4SLinus Torvalds * Fred N. van Kempen, <waltje@uWalt.NL.Mugnet.ORG> 101da177e4SLinus Torvalds * Mark Evans, <evansmp@uhura.aston.ac.uk> 111da177e4SLinus Torvalds * Corey Minyard <wf-rch!minyard@relay.EU.net> 121da177e4SLinus Torvalds * Florian La Roche, <flla@stud.uni-sb.de> 131da177e4SLinus Torvalds * Charles Hedrick, <hedrick@klinzhai.rutgers.edu> 141da177e4SLinus Torvalds * Linus Torvalds, <torvalds@cs.helsinki.fi> 151da177e4SLinus Torvalds * Alan Cox, <gw4pts@gw4pts.ampr.org> 161da177e4SLinus Torvalds * Matthew Dillon, <dillon@apollo.west.oic.com> 171da177e4SLinus Torvalds * Arnt Gulbrandsen, <agulbra@nvg.unit.no> 181da177e4SLinus Torvalds * Jorge Cwik, <jorge@laser.satlink.net> 191da177e4SLinus Torvalds * 201da177e4SLinus Torvalds * Fixes: 211da177e4SLinus Torvalds * Alan Cox : Numerous verify_area() calls 221da177e4SLinus Torvalds * Alan Cox : Set the ACK bit on a reset 231da177e4SLinus Torvalds * Alan Cox : Stopped it crashing if it closed while 241da177e4SLinus Torvalds * sk->inuse=1 and was trying to connect 251da177e4SLinus Torvalds * (tcp_err()). 261da177e4SLinus Torvalds * Alan Cox : All icmp error handling was broken 271da177e4SLinus Torvalds * pointers passed where wrong and the 281da177e4SLinus Torvalds * socket was looked up backwards. Nobody 291da177e4SLinus Torvalds * tested any icmp error code obviously. 301da177e4SLinus Torvalds * Alan Cox : tcp_err() now handled properly. It 311da177e4SLinus Torvalds * wakes people on errors. poll 321da177e4SLinus Torvalds * behaves and the icmp error race 331da177e4SLinus Torvalds * has gone by moving it into sock.c 341da177e4SLinus Torvalds * Alan Cox : tcp_send_reset() fixed to work for 351da177e4SLinus Torvalds * everything not just packets for 361da177e4SLinus Torvalds * unknown sockets. 371da177e4SLinus Torvalds * Alan Cox : tcp option processing. 381da177e4SLinus Torvalds * Alan Cox : Reset tweaked (still not 100%) [Had 391da177e4SLinus Torvalds * syn rule wrong] 401da177e4SLinus Torvalds * Herp Rosmanith : More reset fixes 411da177e4SLinus Torvalds * Alan Cox : No longer acks invalid rst frames. 421da177e4SLinus Torvalds * Acking any kind of RST is right out. 431da177e4SLinus Torvalds * Alan Cox : Sets an ignore me flag on an rst 441da177e4SLinus Torvalds * receive otherwise odd bits of prattle 451da177e4SLinus Torvalds * escape still 461da177e4SLinus Torvalds * Alan Cox : Fixed another acking RST frame bug. 471da177e4SLinus Torvalds * Should stop LAN workplace lockups. 481da177e4SLinus Torvalds * Alan Cox : Some tidyups using the new skb list 491da177e4SLinus Torvalds * facilities 501da177e4SLinus Torvalds * Alan Cox : sk->keepopen now seems to work 511da177e4SLinus Torvalds * Alan Cox : Pulls options out correctly on accepts 521da177e4SLinus Torvalds * Alan Cox : Fixed assorted sk->rqueue->next errors 531da177e4SLinus Torvalds * Alan Cox : PSH doesn't end a TCP read. Switched a 541da177e4SLinus Torvalds * bit to skb ops. 551da177e4SLinus Torvalds * Alan Cox : Tidied tcp_data to avoid a potential 561da177e4SLinus Torvalds * nasty. 571da177e4SLinus Torvalds * Alan Cox : Added some better commenting, as the 581da177e4SLinus Torvalds * tcp is hard to follow 591da177e4SLinus Torvalds * Alan Cox : Removed incorrect check for 20 * psh 601da177e4SLinus Torvalds * Michael O'Reilly : ack < copied bug fix. 611da177e4SLinus Torvalds * Johannes Stille : Misc tcp fixes (not all in yet). 621da177e4SLinus Torvalds * Alan Cox : FIN with no memory -> CRASH 631da177e4SLinus Torvalds * Alan Cox : Added socket option proto entries. 641da177e4SLinus Torvalds * Also added awareness of them to accept. 651da177e4SLinus Torvalds * Alan Cox : Added TCP options (SOL_TCP) 661da177e4SLinus Torvalds * Alan Cox : Switched wakeup calls to callbacks, 671da177e4SLinus Torvalds * so the kernel can layer network 681da177e4SLinus Torvalds * sockets. 691da177e4SLinus Torvalds * Alan Cox : Use ip_tos/ip_ttl settings. 701da177e4SLinus Torvalds * Alan Cox : Handle FIN (more) properly (we hope). 711da177e4SLinus Torvalds * Alan Cox : RST frames sent on unsynchronised 721da177e4SLinus Torvalds * state ack error. 731da177e4SLinus Torvalds * Alan Cox : Put in missing check for SYN bit. 741da177e4SLinus Torvalds * Alan Cox : Added tcp_select_window() aka NET2E 751da177e4SLinus Torvalds * window non shrink trick. 761da177e4SLinus Torvalds * Alan Cox : Added a couple of small NET2E timer 771da177e4SLinus Torvalds * fixes 781da177e4SLinus Torvalds * Charles Hedrick : TCP fixes 791da177e4SLinus Torvalds * Toomas Tamm : TCP window fixes 801da177e4SLinus Torvalds * Alan Cox : Small URG fix to rlogin ^C ack fight 811da177e4SLinus Torvalds * Charles Hedrick : Rewrote most of it to actually work 821da177e4SLinus Torvalds * Linus : Rewrote tcp_read() and URG handling 831da177e4SLinus Torvalds * completely 841da177e4SLinus Torvalds * Gerhard Koerting: Fixed some missing timer handling 851da177e4SLinus Torvalds * Matthew Dillon : Reworked TCP machine states as per RFC 861da177e4SLinus Torvalds * Gerhard Koerting: PC/TCP workarounds 871da177e4SLinus Torvalds * Adam Caldwell : Assorted timer/timing errors 881da177e4SLinus Torvalds * Matthew Dillon : Fixed another RST bug 891da177e4SLinus Torvalds * Alan Cox : Move to kernel side addressing changes. 901da177e4SLinus Torvalds * Alan Cox : Beginning work on TCP fastpathing 911da177e4SLinus Torvalds * (not yet usable) 921da177e4SLinus Torvalds * Arnt Gulbrandsen: Turbocharged tcp_check() routine. 931da177e4SLinus Torvalds * Alan Cox : TCP fast path debugging 941da177e4SLinus Torvalds * Alan Cox : Window clamping 951da177e4SLinus Torvalds * Michael Riepe : Bug in tcp_check() 961da177e4SLinus Torvalds * Matt Dillon : More TCP improvements and RST bug fixes 971da177e4SLinus Torvalds * Matt Dillon : Yet more small nasties remove from the 981da177e4SLinus Torvalds * TCP code (Be very nice to this man if 991da177e4SLinus Torvalds * tcp finally works 100%) 8) 1001da177e4SLinus Torvalds * Alan Cox : BSD accept semantics. 1011da177e4SLinus Torvalds * Alan Cox : Reset on closedown bug. 1021da177e4SLinus Torvalds * Peter De Schrijver : ENOTCONN check missing in tcp_sendto(). 1031da177e4SLinus Torvalds * Michael Pall : Handle poll() after URG properly in 1041da177e4SLinus Torvalds * all cases. 1051da177e4SLinus Torvalds * Michael Pall : Undo the last fix in tcp_read_urg() 1061da177e4SLinus Torvalds * (multi URG PUSH broke rlogin). 1071da177e4SLinus Torvalds * Michael Pall : Fix the multi URG PUSH problem in 1081da177e4SLinus Torvalds * tcp_readable(), poll() after URG 1091da177e4SLinus Torvalds * works now. 1101da177e4SLinus Torvalds * Michael Pall : recv(...,MSG_OOB) never blocks in the 1111da177e4SLinus Torvalds * BSD api. 1121da177e4SLinus Torvalds * Alan Cox : Changed the semantics of sk->socket to 1131da177e4SLinus Torvalds * fix a race and a signal problem with 1141da177e4SLinus Torvalds * accept() and async I/O. 1151da177e4SLinus Torvalds * Alan Cox : Relaxed the rules on tcp_sendto(). 1161da177e4SLinus Torvalds * Yury Shevchuk : Really fixed accept() blocking problem. 1171da177e4SLinus Torvalds * Craig I. Hagan : Allow for BSD compatible TIME_WAIT for 1181da177e4SLinus Torvalds * clients/servers which listen in on 1191da177e4SLinus Torvalds * fixed ports. 1201da177e4SLinus Torvalds * Alan Cox : Cleaned the above up and shrank it to 1211da177e4SLinus Torvalds * a sensible code size. 1221da177e4SLinus Torvalds * Alan Cox : Self connect lockup fix. 1231da177e4SLinus Torvalds * Alan Cox : No connect to multicast. 1241da177e4SLinus Torvalds * Ross Biro : Close unaccepted children on master 1251da177e4SLinus Torvalds * socket close. 1261da177e4SLinus Torvalds * Alan Cox : Reset tracing code. 1271da177e4SLinus Torvalds * Alan Cox : Spurious resets on shutdown. 1281da177e4SLinus Torvalds * Alan Cox : Giant 15 minute/60 second timer error 1291da177e4SLinus Torvalds * Alan Cox : Small whoops in polling before an 1301da177e4SLinus Torvalds * accept. 1311da177e4SLinus Torvalds * Alan Cox : Kept the state trace facility since 1321da177e4SLinus Torvalds * it's handy for debugging. 1331da177e4SLinus Torvalds * Alan Cox : More reset handler fixes. 1341da177e4SLinus Torvalds * Alan Cox : Started rewriting the code based on 1351da177e4SLinus Torvalds * the RFC's for other useful protocol 1361da177e4SLinus Torvalds * references see: Comer, KA9Q NOS, and 1371da177e4SLinus Torvalds * for a reference on the difference 1381da177e4SLinus Torvalds * between specifications and how BSD 1391da177e4SLinus Torvalds * works see the 4.4lite source. 1401da177e4SLinus Torvalds * A.N.Kuznetsov : Don't time wait on completion of tidy 1411da177e4SLinus Torvalds * close. 1421da177e4SLinus Torvalds * Linus Torvalds : Fin/Shutdown & copied_seq changes. 1431da177e4SLinus Torvalds * Linus Torvalds : Fixed BSD port reuse to work first syn 1441da177e4SLinus Torvalds * Alan Cox : Reimplemented timers as per the RFC 1451da177e4SLinus Torvalds * and using multiple timers for sanity. 1461da177e4SLinus Torvalds * Alan Cox : Small bug fixes, and a lot of new 1471da177e4SLinus Torvalds * comments. 1481da177e4SLinus Torvalds * Alan Cox : Fixed dual reader crash by locking 1491da177e4SLinus Torvalds * the buffers (much like datagram.c) 1501da177e4SLinus Torvalds * Alan Cox : Fixed stuck sockets in probe. A probe 1511da177e4SLinus Torvalds * now gets fed up of retrying without 1521da177e4SLinus Torvalds * (even a no space) answer. 1531da177e4SLinus Torvalds * Alan Cox : Extracted closing code better 1541da177e4SLinus Torvalds * Alan Cox : Fixed the closing state machine to 1551da177e4SLinus Torvalds * resemble the RFC. 1561da177e4SLinus Torvalds * Alan Cox : More 'per spec' fixes. 1571da177e4SLinus Torvalds * Jorge Cwik : Even faster checksumming. 1581da177e4SLinus Torvalds * Alan Cox : tcp_data() doesn't ack illegal PSH 1591da177e4SLinus Torvalds * only frames. At least one pc tcp stack 1601da177e4SLinus Torvalds * generates them. 1611da177e4SLinus Torvalds * Alan Cox : Cache last socket. 1621da177e4SLinus Torvalds * Alan Cox : Per route irtt. 1631da177e4SLinus Torvalds * Matt Day : poll()->select() match BSD precisely on error 1641da177e4SLinus Torvalds * Alan Cox : New buffers 1651da177e4SLinus Torvalds * Marc Tamsky : Various sk->prot->retransmits and 1661da177e4SLinus Torvalds * sk->retransmits misupdating fixed. 1671da177e4SLinus Torvalds * Fixed tcp_write_timeout: stuck close, 1681da177e4SLinus Torvalds * and TCP syn retries gets used now. 1691da177e4SLinus Torvalds * Mark Yarvis : In tcp_read_wakeup(), don't send an 1701da177e4SLinus Torvalds * ack if state is TCP_CLOSED. 1711da177e4SLinus Torvalds * Alan Cox : Look up device on a retransmit - routes may 1721da177e4SLinus Torvalds * change. Doesn't yet cope with MSS shrink right 1731da177e4SLinus Torvalds * but it's a start! 1741da177e4SLinus Torvalds * Marc Tamsky : Closing in closing fixes. 1751da177e4SLinus Torvalds * Mike Shaver : RFC1122 verifications. 1761da177e4SLinus Torvalds * Alan Cox : rcv_saddr errors. 1771da177e4SLinus Torvalds * Alan Cox : Block double connect(). 1781da177e4SLinus Torvalds * Alan Cox : Small hooks for enSKIP. 1791da177e4SLinus Torvalds * Alexey Kuznetsov: Path MTU discovery. 1801da177e4SLinus Torvalds * Alan Cox : Support soft errors. 1811da177e4SLinus Torvalds * Alan Cox : Fix MTU discovery pathological case 1821da177e4SLinus Torvalds * when the remote claims no mtu! 1831da177e4SLinus Torvalds * Marc Tamsky : TCP_CLOSE fix. 1841da177e4SLinus Torvalds * Colin (G3TNE) : Send a reset on syn ack replies in 1851da177e4SLinus Torvalds * window but wrong (fixes NT lpd problems) 1861da177e4SLinus Torvalds * Pedro Roque : Better TCP window handling, delayed ack. 1871da177e4SLinus Torvalds * Joerg Reuter : No modification of locked buffers in 1881da177e4SLinus Torvalds * tcp_do_retransmit() 1891da177e4SLinus Torvalds * Eric Schenk : Changed receiver side silly window 1901da177e4SLinus Torvalds * avoidance algorithm to BSD style 1911da177e4SLinus Torvalds * algorithm. This doubles throughput 1921da177e4SLinus Torvalds * against machines running Solaris, 1931da177e4SLinus Torvalds * and seems to result in general 1941da177e4SLinus Torvalds * improvement. 1951da177e4SLinus Torvalds * Stefan Magdalinski : adjusted tcp_readable() to fix FIONREAD 1961da177e4SLinus Torvalds * Willy Konynenberg : Transparent proxying support. 1971da177e4SLinus Torvalds * Mike McLagan : Routing by source 1981da177e4SLinus Torvalds * Keith Owens : Do proper merging with partial SKB's in 1991da177e4SLinus Torvalds * tcp_do_sendmsg to avoid burstiness. 2001da177e4SLinus Torvalds * Eric Schenk : Fix fast close down bug with 2011da177e4SLinus Torvalds * shutdown() followed by close(). 2021da177e4SLinus Torvalds * Andi Kleen : Make poll agree with SIGIO 2031da177e4SLinus Torvalds * Salvatore Sanfilippo : Support SO_LINGER with linger == 1 and 2041da177e4SLinus Torvalds * lingertime == 0 (RFC 793 ABORT Call) 2051da177e4SLinus Torvalds * Hirokazu Takahashi : Use copy_from_user() instead of 2061da177e4SLinus Torvalds * csum_and_copy_from_user() if possible. 2071da177e4SLinus Torvalds * 2081da177e4SLinus Torvalds * This program is free software; you can redistribute it and/or 2091da177e4SLinus Torvalds * modify it under the terms of the GNU General Public License 2101da177e4SLinus Torvalds * as published by the Free Software Foundation; either version 2111da177e4SLinus Torvalds * 2 of the License, or(at your option) any later version. 2121da177e4SLinus Torvalds * 2131da177e4SLinus Torvalds * Description of States: 2141da177e4SLinus Torvalds * 2151da177e4SLinus Torvalds * TCP_SYN_SENT sent a connection request, waiting for ack 2161da177e4SLinus Torvalds * 2171da177e4SLinus Torvalds * TCP_SYN_RECV received a connection request, sent ack, 2181da177e4SLinus Torvalds * waiting for final ack in three-way handshake. 2191da177e4SLinus Torvalds * 2201da177e4SLinus Torvalds * TCP_ESTABLISHED connection established 2211da177e4SLinus Torvalds * 2221da177e4SLinus Torvalds * TCP_FIN_WAIT1 our side has shutdown, waiting to complete 2231da177e4SLinus Torvalds * transmission of remaining buffered data 2241da177e4SLinus Torvalds * 2251da177e4SLinus Torvalds * TCP_FIN_WAIT2 all buffered data sent, waiting for remote 2261da177e4SLinus Torvalds * to shutdown 2271da177e4SLinus Torvalds * 2281da177e4SLinus Torvalds * TCP_CLOSING both sides have shutdown but we still have 2291da177e4SLinus Torvalds * data we have to finish sending 2301da177e4SLinus Torvalds * 2311da177e4SLinus Torvalds * TCP_TIME_WAIT timeout to catch resent junk before entering 2321da177e4SLinus Torvalds * closed, can only be entered from FIN_WAIT2 2331da177e4SLinus Torvalds * or CLOSING. Required because the other end 2341da177e4SLinus Torvalds * may not have gotten our last ACK causing it 2351da177e4SLinus Torvalds * to retransmit the data packet (which we ignore) 2361da177e4SLinus Torvalds * 2371da177e4SLinus Torvalds * TCP_CLOSE_WAIT remote side has shutdown and is waiting for 2381da177e4SLinus Torvalds * us to finish writing our data and to shutdown 2391da177e4SLinus Torvalds * (we have to close() to move on to LAST_ACK) 2401da177e4SLinus Torvalds * 2411da177e4SLinus Torvalds * TCP_LAST_ACK out side has shutdown after remote has 2421da177e4SLinus Torvalds * shutdown. There may still be data in our 2431da177e4SLinus Torvalds * buffer that we have to finish sending 2441da177e4SLinus Torvalds * 2451da177e4SLinus Torvalds * TCP_CLOSE socket is finished 2461da177e4SLinus Torvalds */ 2471da177e4SLinus Torvalds 248afd46503SJoe Perches #define pr_fmt(fmt) "TCP: " fmt 249afd46503SJoe Perches 250cf80e0e4SHerbert Xu #include <crypto/hash.h> 251172589ccSIlpo Järvinen #include <linux/kernel.h> 2521da177e4SLinus Torvalds #include <linux/module.h> 2531da177e4SLinus Torvalds #include <linux/types.h> 2541da177e4SLinus Torvalds #include <linux/fcntl.h> 2551da177e4SLinus Torvalds #include <linux/poll.h> 2566e9250f5SEric Dumazet #include <linux/inet_diag.h> 2571da177e4SLinus Torvalds #include <linux/init.h> 2581da177e4SLinus Torvalds #include <linux/fs.h> 2599c55e01cSJens Axboe #include <linux/skbuff.h> 26081b23b4aSAndrew Morton #include <linux/scatterlist.h> 2619c55e01cSJens Axboe #include <linux/splice.h> 2629c55e01cSJens Axboe #include <linux/net.h> 2639c55e01cSJens Axboe #include <linux/socket.h> 2641da177e4SLinus Torvalds #include <linux/random.h> 2651da177e4SLinus Torvalds #include <linux/bootmem.h> 26657413ebcSMiquel van Smoorenburg #include <linux/highmem.h> 26757413ebcSMiquel van Smoorenburg #include <linux/swap.h> 268b8059eadSDavid S. Miller #include <linux/cache.h> 269f4c50d99SHerbert Xu #include <linux/err.h> 270da5c78c8SWilliam Allen Simpson #include <linux/time.h> 2715a0e3ad6STejun Heo #include <linux/slab.h> 27298aaa913SMike Maloney #include <linux/errqueue.h> 2731da177e4SLinus Torvalds 2741da177e4SLinus Torvalds #include <net/icmp.h> 275cf60af03SYuchung Cheng #include <net/inet_common.h> 2761da177e4SLinus Torvalds #include <net/tcp.h> 2771da177e4SLinus Torvalds #include <net/xfrm.h> 2781da177e4SLinus Torvalds #include <net/ip.h> 2799c55e01cSJens Axboe #include <net/sock.h> 2801da177e4SLinus Torvalds 2817c0f6ba6SLinus Torvalds #include <linux/uaccess.h> 2821da177e4SLinus Torvalds #include <asm/ioctls.h> 283076bb0c8SEliezer Tamir #include <net/busy_poll.h> 2841da177e4SLinus Torvalds 285e8fce239SSong Liu #include <trace/events/tcp.h> 286e8fce239SSong Liu 28795bd09ebSEric Dumazet int sysctl_tcp_min_tso_segs __read_mostly = 2; 28895bd09ebSEric Dumazet 289f54b3111SEric Dumazet int sysctl_tcp_autocorking __read_mostly = 1; 290f54b3111SEric Dumazet 291dd24c001SEric Dumazet struct percpu_counter tcp_orphan_count; 2920a5578cfSArnaldo Carvalho de Melo EXPORT_SYMBOL_GPL(tcp_orphan_count); 2930a5578cfSArnaldo Carvalho de Melo 294a4fe34bfSEric W. Biederman long sysctl_tcp_mem[3] __read_mostly; 295b8059eadSDavid S. Miller int sysctl_tcp_wmem[3] __read_mostly; 296b8059eadSDavid S. Miller int sysctl_tcp_rmem[3] __read_mostly; 2971da177e4SLinus Torvalds 298a4fe34bfSEric W. Biederman EXPORT_SYMBOL(sysctl_tcp_mem); 2991da177e4SLinus Torvalds EXPORT_SYMBOL(sysctl_tcp_rmem); 3001da177e4SLinus Torvalds EXPORT_SYMBOL(sysctl_tcp_wmem); 3011da177e4SLinus Torvalds 3028d987e5cSEric Dumazet atomic_long_t tcp_memory_allocated; /* Current allocated memory. */ 3031da177e4SLinus Torvalds EXPORT_SYMBOL(tcp_memory_allocated); 3041748376bSEric Dumazet 3051748376bSEric Dumazet /* 3061748376bSEric Dumazet * Current number of TCP sockets. 3071748376bSEric Dumazet */ 3081748376bSEric Dumazet struct percpu_counter tcp_sockets_allocated; 3091da177e4SLinus Torvalds EXPORT_SYMBOL(tcp_sockets_allocated); 3101da177e4SLinus Torvalds 3111da177e4SLinus Torvalds /* 3129c55e01cSJens Axboe * TCP splice context 3139c55e01cSJens Axboe */ 3149c55e01cSJens Axboe struct tcp_splice_state { 3159c55e01cSJens Axboe struct pipe_inode_info *pipe; 3169c55e01cSJens Axboe size_t len; 3179c55e01cSJens Axboe unsigned int flags; 3189c55e01cSJens Axboe }; 3199c55e01cSJens Axboe 3209c55e01cSJens Axboe /* 3211da177e4SLinus Torvalds * Pressure flag: try to collapse. 3221da177e4SLinus Torvalds * Technical note: it is used by multiple contexts non atomically. 3233ab224beSHideo Aoki * All the __sk_mem_schedule() is of this nature: accounting 3241da177e4SLinus Torvalds * is strict, actions are advisory and have some latency. 3251da177e4SLinus Torvalds */ 32606044751SEric Dumazet unsigned long tcp_memory_pressure __read_mostly; 32706044751SEric Dumazet EXPORT_SYMBOL_GPL(tcp_memory_pressure); 3281da177e4SLinus Torvalds 3295c52ba17SPavel Emelyanov void tcp_enter_memory_pressure(struct sock *sk) 3301da177e4SLinus Torvalds { 33106044751SEric Dumazet unsigned long val; 33206044751SEric Dumazet 33306044751SEric Dumazet if (tcp_memory_pressure) 33406044751SEric Dumazet return; 33506044751SEric Dumazet val = jiffies; 33606044751SEric Dumazet 33706044751SEric Dumazet if (!val) 33806044751SEric Dumazet val--; 33906044751SEric Dumazet if (!cmpxchg(&tcp_memory_pressure, 0, val)) 3404e673444SPavel Emelyanov NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPMEMORYPRESSURES); 3411da177e4SLinus Torvalds } 34206044751SEric Dumazet EXPORT_SYMBOL_GPL(tcp_enter_memory_pressure); 34306044751SEric Dumazet 34406044751SEric Dumazet void tcp_leave_memory_pressure(struct sock *sk) 34506044751SEric Dumazet { 34606044751SEric Dumazet unsigned long val; 34706044751SEric Dumazet 34806044751SEric Dumazet if (!tcp_memory_pressure) 34906044751SEric Dumazet return; 35006044751SEric Dumazet val = xchg(&tcp_memory_pressure, 0); 35106044751SEric Dumazet if (val) 35206044751SEric Dumazet NET_ADD_STATS(sock_net(sk), LINUX_MIB_TCPMEMORYPRESSURESCHRONO, 35306044751SEric Dumazet jiffies_to_msecs(jiffies - val)); 3541da177e4SLinus Torvalds } 35506044751SEric Dumazet EXPORT_SYMBOL_GPL(tcp_leave_memory_pressure); 3561da177e4SLinus Torvalds 357b103cf34SJulian Anastasov /* Convert seconds to retransmits based on initial and max timeout */ 358b103cf34SJulian Anastasov static u8 secs_to_retrans(int seconds, int timeout, int rto_max) 359b103cf34SJulian Anastasov { 360b103cf34SJulian Anastasov u8 res = 0; 361b103cf34SJulian Anastasov 362b103cf34SJulian Anastasov if (seconds > 0) { 363b103cf34SJulian Anastasov int period = timeout; 364b103cf34SJulian Anastasov 365b103cf34SJulian Anastasov res = 1; 366b103cf34SJulian Anastasov while (seconds > period && res < 255) { 367b103cf34SJulian Anastasov res++; 368b103cf34SJulian Anastasov timeout <<= 1; 369b103cf34SJulian Anastasov if (timeout > rto_max) 370b103cf34SJulian Anastasov timeout = rto_max; 371b103cf34SJulian Anastasov period += timeout; 372b103cf34SJulian Anastasov } 373b103cf34SJulian Anastasov } 374b103cf34SJulian Anastasov return res; 375b103cf34SJulian Anastasov } 376b103cf34SJulian Anastasov 377b103cf34SJulian Anastasov /* Convert retransmits to seconds based on initial and max timeout */ 378b103cf34SJulian Anastasov static int retrans_to_secs(u8 retrans, int timeout, int rto_max) 379b103cf34SJulian Anastasov { 380b103cf34SJulian Anastasov int period = 0; 381b103cf34SJulian Anastasov 382b103cf34SJulian Anastasov if (retrans > 0) { 383b103cf34SJulian Anastasov period = timeout; 384b103cf34SJulian Anastasov while (--retrans) { 385b103cf34SJulian Anastasov timeout <<= 1; 386b103cf34SJulian Anastasov if (timeout > rto_max) 387b103cf34SJulian Anastasov timeout = rto_max; 388b103cf34SJulian Anastasov period += timeout; 389b103cf34SJulian Anastasov } 390b103cf34SJulian Anastasov } 391b103cf34SJulian Anastasov return period; 392b103cf34SJulian Anastasov } 393b103cf34SJulian Anastasov 3940263598cSWei Wang static u64 tcp_compute_delivery_rate(const struct tcp_sock *tp) 3950263598cSWei Wang { 3960263598cSWei Wang u32 rate = READ_ONCE(tp->rate_delivered); 3970263598cSWei Wang u32 intv = READ_ONCE(tp->rate_interval_us); 3980263598cSWei Wang u64 rate64 = 0; 3990263598cSWei Wang 4000263598cSWei Wang if (rate && intv) { 4010263598cSWei Wang rate64 = (u64)rate * tp->mss_cache * USEC_PER_SEC; 4020263598cSWei Wang do_div(rate64, intv); 4030263598cSWei Wang } 4040263598cSWei Wang return rate64; 4050263598cSWei Wang } 4060263598cSWei Wang 407900f65d3SNeal Cardwell /* Address-family independent initialization for a tcp_sock. 408900f65d3SNeal Cardwell * 409900f65d3SNeal Cardwell * NOTE: A lot of things set to zero explicitly by call to 410900f65d3SNeal Cardwell * sk_alloc() so need not be done here. 411900f65d3SNeal Cardwell */ 412900f65d3SNeal Cardwell void tcp_init_sock(struct sock *sk) 413900f65d3SNeal Cardwell { 414900f65d3SNeal Cardwell struct inet_connection_sock *icsk = inet_csk(sk); 415900f65d3SNeal Cardwell struct tcp_sock *tp = tcp_sk(sk); 416900f65d3SNeal Cardwell 4179f5afeaeSYaogong Wang tp->out_of_order_queue = RB_ROOT; 41875c119afSEric Dumazet sk->tcp_rtx_queue = RB_ROOT; 419900f65d3SNeal Cardwell tcp_init_xmit_timers(sk); 42046d3ceabSEric Dumazet INIT_LIST_HEAD(&tp->tsq_node); 421e2080072SEric Dumazet INIT_LIST_HEAD(&tp->tsorted_sent_queue); 422900f65d3SNeal Cardwell 423900f65d3SNeal Cardwell icsk->icsk_rto = TCP_TIMEOUT_INIT; 424740b0f18SEric Dumazet tp->mdev_us = jiffies_to_usecs(TCP_TIMEOUT_INIT); 425ac9517fcSEric Dumazet minmax_reset(&tp->rtt_min, tcp_jiffies32, ~0U); 426900f65d3SNeal Cardwell 427900f65d3SNeal Cardwell /* So many TCP implementations out there (incorrectly) count the 428900f65d3SNeal Cardwell * initial SYN frame in their delayed-ACK and congestion control 429900f65d3SNeal Cardwell * algorithms that we must have the following bandaid to talk 430900f65d3SNeal Cardwell * efficiently to them. -DaveM 431900f65d3SNeal Cardwell */ 432900f65d3SNeal Cardwell tp->snd_cwnd = TCP_INIT_CWND; 433900f65d3SNeal Cardwell 434d7722e85SSoheil Hassas Yeganeh /* There's a bubble in the pipe until at least the first ACK. */ 435d7722e85SSoheil Hassas Yeganeh tp->app_limited = ~0U; 436d7722e85SSoheil Hassas Yeganeh 437900f65d3SNeal Cardwell /* See draft-stevens-tcpca-spec-01 for discussion of the 438900f65d3SNeal Cardwell * initialization of these values. 439900f65d3SNeal Cardwell */ 440900f65d3SNeal Cardwell tp->snd_ssthresh = TCP_INFINITE_SSTHRESH; 441900f65d3SNeal Cardwell tp->snd_cwnd_clamp = ~0; 442900f65d3SNeal Cardwell tp->mss_cache = TCP_MSS_DEFAULT; 443900f65d3SNeal Cardwell 4441043e25fSNikolay Borisov tp->reordering = sock_net(sk)->ipv4.sysctl_tcp_reordering; 44555d8694fSFlorian Westphal tcp_assign_congestion_control(sk); 446900f65d3SNeal Cardwell 447ceaa1fefSAndrey Vagin tp->tsoffset = 0; 448ceaa1fefSAndrey Vagin 449900f65d3SNeal Cardwell sk->sk_state = TCP_CLOSE; 450900f65d3SNeal Cardwell 451900f65d3SNeal Cardwell sk->sk_write_space = sk_stream_write_space; 452900f65d3SNeal Cardwell sock_set_flag(sk, SOCK_USE_WRITE_QUEUE); 453900f65d3SNeal Cardwell 454900f65d3SNeal Cardwell icsk->icsk_sync_mss = tcp_sync_mss; 455900f65d3SNeal Cardwell 456900f65d3SNeal Cardwell sk->sk_sndbuf = sysctl_tcp_wmem[1]; 457900f65d3SNeal Cardwell sk->sk_rcvbuf = sysctl_tcp_rmem[1]; 458900f65d3SNeal Cardwell 459900f65d3SNeal Cardwell sk_sockets_allocated_inc(sk); 460900f65d3SNeal Cardwell } 461900f65d3SNeal Cardwell EXPORT_SYMBOL(tcp_init_sock); 462900f65d3SNeal Cardwell 46327204aaaSWei Wang void tcp_init_transfer(struct sock *sk, int bpf_op) 46427204aaaSWei Wang { 46527204aaaSWei Wang struct inet_connection_sock *icsk = inet_csk(sk); 46627204aaaSWei Wang 46727204aaaSWei Wang tcp_mtup_init(sk); 46827204aaaSWei Wang icsk->icsk_af_ops->rebuild_header(sk); 46927204aaaSWei Wang tcp_init_metrics(sk); 47027204aaaSWei Wang tcp_call_bpf(sk, bpf_op); 47127204aaaSWei Wang tcp_init_congestion_control(sk); 47227204aaaSWei Wang tcp_init_buffer_space(sk); 47327204aaaSWei Wang } 47427204aaaSWei Wang 4754e8cc228SEric Dumazet static void tcp_tx_timestamp(struct sock *sk, u16 tsflags) 4764ed2d765SWillem de Bruijn { 4774e8cc228SEric Dumazet struct sk_buff *skb = tcp_write_queue_tail(sk); 4784e8cc228SEric Dumazet 479ad02c4f5SSoheil Hassas Yeganeh if (tsflags && skb) { 4804ed2d765SWillem de Bruijn struct skb_shared_info *shinfo = skb_shinfo(skb); 4816b084928SSoheil Hassas Yeganeh struct tcp_skb_cb *tcb = TCP_SKB_CB(skb); 4824ed2d765SWillem de Bruijn 483c14ac945SSoheil Hassas Yeganeh sock_tx_timestamp(sk, tsflags, &shinfo->tx_flags); 4840a2cf20cSSoheil Hassas Yeganeh if (tsflags & SOF_TIMESTAMPING_TX_ACK) 4850a2cf20cSSoheil Hassas Yeganeh tcb->txstamp_ack = 1; 4860a2cf20cSSoheil Hassas Yeganeh if (tsflags & SOF_TIMESTAMPING_TX_RECORD_MASK) 4874ed2d765SWillem de Bruijn shinfo->tskey = TCP_SKB_CB(skb)->seq + skb->len - 1; 4884ed2d765SWillem de Bruijn } 489f066e2b0SWillem de Bruijn } 4904ed2d765SWillem de Bruijn 4911da177e4SLinus Torvalds /* 4921da177e4SLinus Torvalds * Wait for a TCP event. 4931da177e4SLinus Torvalds * 4941da177e4SLinus Torvalds * Note that we don't need to lock the socket, as the upper poll layers 4951da177e4SLinus Torvalds * take care of normal races (between the test and the event) and we don't 4961da177e4SLinus Torvalds * go look at any of the socket buffers directly. 4971da177e4SLinus Torvalds */ 4981da177e4SLinus Torvalds unsigned int tcp_poll(struct file *file, struct socket *sock, poll_table *wait) 4991da177e4SLinus Torvalds { 5001da177e4SLinus Torvalds unsigned int mask; 5011da177e4SLinus Torvalds struct sock *sk = sock->sk; 502cf533ea5SEric Dumazet const struct tcp_sock *tp = tcp_sk(sk); 50300fd38d9SEric Dumazet int state; 5041da177e4SLinus Torvalds 505c3f1dbafSDavid Majnemer sock_rps_record_flow(sk); 506c3f1dbafSDavid Majnemer 507aa395145SEric Dumazet sock_poll_wait(file, sk_sleep(sk), wait); 50800fd38d9SEric Dumazet 50900fd38d9SEric Dumazet state = sk_state_load(sk); 51000fd38d9SEric Dumazet if (state == TCP_LISTEN) 511dc40c7bcSArnaldo Carvalho de Melo return inet_csk_listen_poll(sk); 5121da177e4SLinus Torvalds 5131da177e4SLinus Torvalds /* Socket is not locked. We are protected from async events 51470efce27SWill Newton * by poll logic and correct handling of state changes 51570efce27SWill Newton * made by other threads is impossible in any case. 5161da177e4SLinus Torvalds */ 5171da177e4SLinus Torvalds 5181da177e4SLinus Torvalds mask = 0; 5191da177e4SLinus Torvalds 5201da177e4SLinus Torvalds /* 5211da177e4SLinus Torvalds * POLLHUP is certainly not done right. But poll() doesn't 5221da177e4SLinus Torvalds * have a notion of HUP in just one direction, and for a 5231da177e4SLinus Torvalds * socket the read side is more interesting. 5241da177e4SLinus Torvalds * 5251da177e4SLinus Torvalds * Some poll() documentation says that POLLHUP is incompatible 5261da177e4SLinus Torvalds * with the POLLOUT/POLLWR flags, so somebody should check this 5271da177e4SLinus Torvalds * all. But careful, it tends to be safer to return too many 5281da177e4SLinus Torvalds * bits than too few, and you can easily break real applications 5291da177e4SLinus Torvalds * if you don't tell them that something has hung up! 5301da177e4SLinus Torvalds * 5311da177e4SLinus Torvalds * Check-me. 5321da177e4SLinus Torvalds * 5331da177e4SLinus Torvalds * Check number 1. POLLHUP is _UNMASKABLE_ event (see UNIX98 and 5341da177e4SLinus Torvalds * our fs/select.c). It means that after we received EOF, 5351da177e4SLinus Torvalds * poll always returns immediately, making impossible poll() on write() 5361da177e4SLinus Torvalds * in state CLOSE_WAIT. One solution is evident --- to set POLLHUP 5371da177e4SLinus Torvalds * if and only if shutdown has been made in both directions. 5381da177e4SLinus Torvalds * Actually, it is interesting to look how Solaris and DUX 53970efce27SWill Newton * solve this dilemma. I would prefer, if POLLHUP were maskable, 5401da177e4SLinus Torvalds * then we could set it on SND_SHUTDOWN. BTW examples given 5411da177e4SLinus Torvalds * in Stevens' books assume exactly this behaviour, it explains 54270efce27SWill Newton * why POLLHUP is incompatible with POLLOUT. --ANK 5431da177e4SLinus Torvalds * 5441da177e4SLinus Torvalds * NOTE. Check for TCP_CLOSE is added. The goal is to prevent 5451da177e4SLinus Torvalds * blocking on fresh not-connected or disconnected socket. --ANK 5461da177e4SLinus Torvalds */ 54700fd38d9SEric Dumazet if (sk->sk_shutdown == SHUTDOWN_MASK || state == TCP_CLOSE) 5481da177e4SLinus Torvalds mask |= POLLHUP; 5491da177e4SLinus Torvalds if (sk->sk_shutdown & RCV_SHUTDOWN) 550f348d70aSDavide Libenzi mask |= POLLIN | POLLRDNORM | POLLRDHUP; 5511da177e4SLinus Torvalds 5528336886fSJerry Chu /* Connected or passive Fast Open socket? */ 55300fd38d9SEric Dumazet if (state != TCP_SYN_SENT && 55400fd38d9SEric Dumazet (state != TCP_SYN_RECV || tp->fastopen_rsk)) { 555c7004482SDavid S. Miller int target = sock_rcvlowat(sk, 0, INT_MAX); 556c7004482SDavid S. Miller 557c7004482SDavid S. Miller if (tp->urg_seq == tp->copied_seq && 558c7004482SDavid S. Miller !sock_flag(sk, SOCK_URGINLINE) && 559c7004482SDavid S. Miller tp->urg_data) 560b634f875SAlexandra Kossovsky target++; 561c7004482SDavid S. Miller 562c7004482SDavid S. Miller if (tp->rcv_nxt - tp->copied_seq >= target) 5631da177e4SLinus Torvalds mask |= POLLIN | POLLRDNORM; 5641da177e4SLinus Torvalds 5651da177e4SLinus Torvalds if (!(sk->sk_shutdown & SEND_SHUTDOWN)) { 56664dc6130SEric Dumazet if (sk_stream_is_writeable(sk)) { 5671da177e4SLinus Torvalds mask |= POLLOUT | POLLWRNORM; 5681da177e4SLinus Torvalds } else { /* send SIGIO later */ 5699cd3e072SEric Dumazet sk_set_bit(SOCKWQ_ASYNC_NOSPACE, sk); 5701da177e4SLinus Torvalds set_bit(SOCK_NOSPACE, &sk->sk_socket->flags); 5711da177e4SLinus Torvalds 5721da177e4SLinus Torvalds /* Race breaker. If space is freed after 5731da177e4SLinus Torvalds * wspace test but before the flags are set, 5743c715127Sjbaron@akamai.com * IO signal will be lost. Memory barrier 5753c715127Sjbaron@akamai.com * pairs with the input side. 5761da177e4SLinus Torvalds */ 5773c715127Sjbaron@akamai.com smp_mb__after_atomic(); 57864dc6130SEric Dumazet if (sk_stream_is_writeable(sk)) 5791da177e4SLinus Torvalds mask |= POLLOUT | POLLWRNORM; 5801da177e4SLinus Torvalds } 581d84ba638SKOSAKI Motohiro } else 582d84ba638SKOSAKI Motohiro mask |= POLLOUT | POLLWRNORM; 5831da177e4SLinus Torvalds 5841da177e4SLinus Torvalds if (tp->urg_data & TCP_URG_VALID) 5851da177e4SLinus Torvalds mask |= POLLPRI; 586d68be71eSDavide Caratti } else if (state == TCP_SYN_SENT && inet_sk(sk)->defer_connect) { 58719f6d3f3SWei Wang /* Active TCP fastopen socket with defer_connect 58819f6d3f3SWei Wang * Return POLLOUT so application can call write() 58919f6d3f3SWei Wang * in order for kernel to generate SYN+data 59019f6d3f3SWei Wang */ 59119f6d3f3SWei Wang mask |= POLLOUT | POLLWRNORM; 5921da177e4SLinus Torvalds } 593a4d25803STom Marshall /* This barrier is coupled with smp_wmb() in tcp_reset() */ 594a4d25803STom Marshall smp_rmb(); 5954ed2d765SWillem de Bruijn if (sk->sk_err || !skb_queue_empty(&sk->sk_error_queue)) 596a4d25803STom Marshall mask |= POLLERR; 597a4d25803STom Marshall 5981da177e4SLinus Torvalds return mask; 5991da177e4SLinus Torvalds } 6004bc2f18bSEric Dumazet EXPORT_SYMBOL(tcp_poll); 6011da177e4SLinus Torvalds 6021da177e4SLinus Torvalds int tcp_ioctl(struct sock *sk, int cmd, unsigned long arg) 6031da177e4SLinus Torvalds { 6041da177e4SLinus Torvalds struct tcp_sock *tp = tcp_sk(sk); 6051da177e4SLinus Torvalds int answ; 6060e71c55cSEric Dumazet bool slow; 6071da177e4SLinus Torvalds 6081da177e4SLinus Torvalds switch (cmd) { 6091da177e4SLinus Torvalds case SIOCINQ: 6101da177e4SLinus Torvalds if (sk->sk_state == TCP_LISTEN) 6111da177e4SLinus Torvalds return -EINVAL; 6121da177e4SLinus Torvalds 6130e71c55cSEric Dumazet slow = lock_sock_fast(sk); 614473bd239STom Herbert answ = tcp_inq(sk); 6150e71c55cSEric Dumazet unlock_sock_fast(sk, slow); 6161da177e4SLinus Torvalds break; 6171da177e4SLinus Torvalds case SIOCATMARK: 6181da177e4SLinus Torvalds answ = tp->urg_data && tp->urg_seq == tp->copied_seq; 6191da177e4SLinus Torvalds break; 6201da177e4SLinus Torvalds case SIOCOUTQ: 6211da177e4SLinus Torvalds if (sk->sk_state == TCP_LISTEN) 6221da177e4SLinus Torvalds return -EINVAL; 6231da177e4SLinus Torvalds 6241da177e4SLinus Torvalds if ((1 << sk->sk_state) & (TCPF_SYN_SENT | TCPF_SYN_RECV)) 6251da177e4SLinus Torvalds answ = 0; 6261da177e4SLinus Torvalds else 6271da177e4SLinus Torvalds answ = tp->write_seq - tp->snd_una; 6281da177e4SLinus Torvalds break; 6292f4e1b39SMario Schuknecht case SIOCOUTQNSD: 6302f4e1b39SMario Schuknecht if (sk->sk_state == TCP_LISTEN) 6312f4e1b39SMario Schuknecht return -EINVAL; 6322f4e1b39SMario Schuknecht 6332f4e1b39SMario Schuknecht if ((1 << sk->sk_state) & (TCPF_SYN_SENT | TCPF_SYN_RECV)) 6342f4e1b39SMario Schuknecht answ = 0; 6352f4e1b39SMario Schuknecht else 6362f4e1b39SMario Schuknecht answ = tp->write_seq - tp->snd_nxt; 6372f4e1b39SMario Schuknecht break; 6381da177e4SLinus Torvalds default: 6391da177e4SLinus Torvalds return -ENOIOCTLCMD; 6403ff50b79SStephen Hemminger } 6411da177e4SLinus Torvalds 6421da177e4SLinus Torvalds return put_user(answ, (int __user *)arg); 6431da177e4SLinus Torvalds } 6444bc2f18bSEric Dumazet EXPORT_SYMBOL(tcp_ioctl); 6451da177e4SLinus Torvalds 6461da177e4SLinus Torvalds static inline void tcp_mark_push(struct tcp_sock *tp, struct sk_buff *skb) 6471da177e4SLinus Torvalds { 6484de075e0SEric Dumazet TCP_SKB_CB(skb)->tcp_flags |= TCPHDR_PSH; 6491da177e4SLinus Torvalds tp->pushed_seq = tp->write_seq; 6501da177e4SLinus Torvalds } 6511da177e4SLinus Torvalds 652a2a385d6SEric Dumazet static inline bool forced_push(const struct tcp_sock *tp) 6531da177e4SLinus Torvalds { 6541da177e4SLinus Torvalds return after(tp->write_seq, tp->pushed_seq + (tp->max_window >> 1)); 6551da177e4SLinus Torvalds } 6561da177e4SLinus Torvalds 657f4a775d1SEric Dumazet static void skb_entail(struct sock *sk, struct sk_buff *skb) 6581da177e4SLinus Torvalds { 6599e412ba7SIlpo Järvinen struct tcp_sock *tp = tcp_sk(sk); 660352d4800SArnaldo Carvalho de Melo struct tcp_skb_cb *tcb = TCP_SKB_CB(skb); 661352d4800SArnaldo Carvalho de Melo 6621da177e4SLinus Torvalds skb->csum = 0; 663352d4800SArnaldo Carvalho de Melo tcb->seq = tcb->end_seq = tp->write_seq; 6644de075e0SEric Dumazet tcb->tcp_flags = TCPHDR_ACK; 665352d4800SArnaldo Carvalho de Melo tcb->sacked = 0; 666f4a775d1SEric Dumazet __skb_header_release(skb); 667fe067e8aSDavid S. Miller tcp_add_write_queue_tail(sk, skb); 6683ab224beSHideo Aoki sk->sk_wmem_queued += skb->truesize; 6693ab224beSHideo Aoki sk_mem_charge(sk, skb->truesize); 67089ebd197SDavid S. Miller if (tp->nonagle & TCP_NAGLE_PUSH) 6711da177e4SLinus Torvalds tp->nonagle &= ~TCP_NAGLE_PUSH; 6726f021c62SEric Dumazet 6736f021c62SEric Dumazet tcp_slow_start_after_idle_check(sk); 6741da177e4SLinus Torvalds } 6751da177e4SLinus Torvalds 676afeca340SKrishna Kumar static inline void tcp_mark_urg(struct tcp_sock *tp, int flags) 6771da177e4SLinus Torvalds { 67833f5f57eSIlpo Järvinen if (flags & MSG_OOB) 6791da177e4SLinus Torvalds tp->snd_up = tp->write_seq; 6801da177e4SLinus Torvalds } 6811da177e4SLinus Torvalds 682f54b3111SEric Dumazet /* If a not yet filled skb is pushed, do not send it if 683a181ceb5SEric Dumazet * we have data packets in Qdisc or NIC queues : 684f54b3111SEric Dumazet * Because TX completion will happen shortly, it gives a chance 685f54b3111SEric Dumazet * to coalesce future sendmsg() payload into this skb, without 686f54b3111SEric Dumazet * need for a timer, and with no latency trade off. 687f54b3111SEric Dumazet * As packets containing data payload have a bigger truesize 688a181ceb5SEric Dumazet * than pure acks (dataless) packets, the last checks prevent 689a181ceb5SEric Dumazet * autocorking if we only have an ACK in Qdisc/NIC queues, 690a181ceb5SEric Dumazet * or if TX completion was delayed after we processed ACK packet. 691f54b3111SEric Dumazet */ 692f54b3111SEric Dumazet static bool tcp_should_autocork(struct sock *sk, struct sk_buff *skb, 693f54b3111SEric Dumazet int size_goal) 6941da177e4SLinus Torvalds { 695f54b3111SEric Dumazet return skb->len < size_goal && 696f54b3111SEric Dumazet sysctl_tcp_autocorking && 697a181ceb5SEric Dumazet skb != tcp_write_queue_head(sk) && 69814afee4bSReshetova, Elena refcount_read(&sk->sk_wmem_alloc) > skb->truesize; 699f54b3111SEric Dumazet } 7009e412ba7SIlpo Järvinen 701f54b3111SEric Dumazet static void tcp_push(struct sock *sk, int flags, int mss_now, 702f54b3111SEric Dumazet int nonagle, int size_goal) 703f54b3111SEric Dumazet { 704f54b3111SEric Dumazet struct tcp_sock *tp = tcp_sk(sk); 705f54b3111SEric Dumazet struct sk_buff *skb; 706f54b3111SEric Dumazet 707f54b3111SEric Dumazet skb = tcp_write_queue_tail(sk); 70875c119afSEric Dumazet if (!skb) 70975c119afSEric Dumazet return; 7101da177e4SLinus Torvalds if (!(flags & MSG_MORE) || forced_push(tp)) 711f54b3111SEric Dumazet tcp_mark_push(tp, skb); 712afeca340SKrishna Kumar 713afeca340SKrishna Kumar tcp_mark_urg(tp, flags); 714f54b3111SEric Dumazet 715f54b3111SEric Dumazet if (tcp_should_autocork(sk, skb, size_goal)) { 716f54b3111SEric Dumazet 717f54b3111SEric Dumazet /* avoid atomic op if TSQ_THROTTLED bit is already set */ 7187aa5470cSEric Dumazet if (!test_bit(TSQ_THROTTLED, &sk->sk_tsq_flags)) { 719f54b3111SEric Dumazet NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPAUTOCORKING); 7207aa5470cSEric Dumazet set_bit(TSQ_THROTTLED, &sk->sk_tsq_flags); 7211da177e4SLinus Torvalds } 722a181ceb5SEric Dumazet /* It is possible TX completion already happened 723a181ceb5SEric Dumazet * before we set TSQ_THROTTLED. 724a181ceb5SEric Dumazet */ 72514afee4bSReshetova, Elena if (refcount_read(&sk->sk_wmem_alloc) > skb->truesize) 726f54b3111SEric Dumazet return; 727f54b3111SEric Dumazet } 728f54b3111SEric Dumazet 729f54b3111SEric Dumazet if (flags & MSG_MORE) 730f54b3111SEric Dumazet nonagle = TCP_NAGLE_CORK; 731f54b3111SEric Dumazet 732f54b3111SEric Dumazet __tcp_push_pending_frames(sk, mss_now, nonagle); 7331da177e4SLinus Torvalds } 7341da177e4SLinus Torvalds 7356ff7751dSAdrian Bunk static int tcp_splice_data_recv(read_descriptor_t *rd_desc, struct sk_buff *skb, 7369c55e01cSJens Axboe unsigned int offset, size_t len) 7379c55e01cSJens Axboe { 7389c55e01cSJens Axboe struct tcp_splice_state *tss = rd_desc->arg.data; 73933966dd0SWilly Tarreau int ret; 7409c55e01cSJens Axboe 741a60e3cc7SHannes Frederic Sowa ret = skb_splice_bits(skb, skb->sk, offset, tss->pipe, 74225869262SAl Viro min(rd_desc->count, len), tss->flags); 74333966dd0SWilly Tarreau if (ret > 0) 74433966dd0SWilly Tarreau rd_desc->count -= ret; 74533966dd0SWilly Tarreau return ret; 7469c55e01cSJens Axboe } 7479c55e01cSJens Axboe 7489c55e01cSJens Axboe static int __tcp_splice_read(struct sock *sk, struct tcp_splice_state *tss) 7499c55e01cSJens Axboe { 7509c55e01cSJens Axboe /* Store TCP splice context information in read_descriptor_t. */ 7519c55e01cSJens Axboe read_descriptor_t rd_desc = { 7529c55e01cSJens Axboe .arg.data = tss, 75333966dd0SWilly Tarreau .count = tss->len, 7549c55e01cSJens Axboe }; 7559c55e01cSJens Axboe 7569c55e01cSJens Axboe return tcp_read_sock(sk, &rd_desc, tcp_splice_data_recv); 7579c55e01cSJens Axboe } 7589c55e01cSJens Axboe 7599c55e01cSJens Axboe /** 7609c55e01cSJens Axboe * tcp_splice_read - splice data from TCP socket to a pipe 7619c55e01cSJens Axboe * @sock: socket to splice from 7629c55e01cSJens Axboe * @ppos: position (not valid) 7639c55e01cSJens Axboe * @pipe: pipe to splice to 7649c55e01cSJens Axboe * @len: number of bytes to splice 7659c55e01cSJens Axboe * @flags: splice modifier flags 7669c55e01cSJens Axboe * 7679c55e01cSJens Axboe * Description: 7689c55e01cSJens Axboe * Will read pages from given socket and fill them into a pipe. 7699c55e01cSJens Axboe * 7709c55e01cSJens Axboe **/ 7719c55e01cSJens Axboe ssize_t tcp_splice_read(struct socket *sock, loff_t *ppos, 7729c55e01cSJens Axboe struct pipe_inode_info *pipe, size_t len, 7739c55e01cSJens Axboe unsigned int flags) 7749c55e01cSJens Axboe { 7759c55e01cSJens Axboe struct sock *sk = sock->sk; 7769c55e01cSJens Axboe struct tcp_splice_state tss = { 7779c55e01cSJens Axboe .pipe = pipe, 7789c55e01cSJens Axboe .len = len, 7799c55e01cSJens Axboe .flags = flags, 7809c55e01cSJens Axboe }; 7819c55e01cSJens Axboe long timeo; 7829c55e01cSJens Axboe ssize_t spliced; 7839c55e01cSJens Axboe int ret; 7849c55e01cSJens Axboe 7853a047bf8SChangli Gao sock_rps_record_flow(sk); 7869c55e01cSJens Axboe /* 7879c55e01cSJens Axboe * We can't seek on a socket input 7889c55e01cSJens Axboe */ 7899c55e01cSJens Axboe if (unlikely(*ppos)) 7909c55e01cSJens Axboe return -ESPIPE; 7919c55e01cSJens Axboe 7929c55e01cSJens Axboe ret = spliced = 0; 7939c55e01cSJens Axboe 7949c55e01cSJens Axboe lock_sock(sk); 7959c55e01cSJens Axboe 79642324c62SEric Dumazet timeo = sock_rcvtimeo(sk, sock->file->f_flags & O_NONBLOCK); 7979c55e01cSJens Axboe while (tss.len) { 7989c55e01cSJens Axboe ret = __tcp_splice_read(sk, &tss); 7999c55e01cSJens Axboe if (ret < 0) 8009c55e01cSJens Axboe break; 8019c55e01cSJens Axboe else if (!ret) { 8029c55e01cSJens Axboe if (spliced) 8039c55e01cSJens Axboe break; 8049c55e01cSJens Axboe if (sock_flag(sk, SOCK_DONE)) 8059c55e01cSJens Axboe break; 8069c55e01cSJens Axboe if (sk->sk_err) { 8079c55e01cSJens Axboe ret = sock_error(sk); 8089c55e01cSJens Axboe break; 8099c55e01cSJens Axboe } 8109c55e01cSJens Axboe if (sk->sk_shutdown & RCV_SHUTDOWN) 8119c55e01cSJens Axboe break; 8129c55e01cSJens Axboe if (sk->sk_state == TCP_CLOSE) { 8139c55e01cSJens Axboe /* 8149c55e01cSJens Axboe * This occurs when user tries to read 8159c55e01cSJens Axboe * from never connected socket. 8169c55e01cSJens Axboe */ 8179c55e01cSJens Axboe if (!sock_flag(sk, SOCK_DONE)) 8189c55e01cSJens Axboe ret = -ENOTCONN; 8199c55e01cSJens Axboe break; 8209c55e01cSJens Axboe } 8219c55e01cSJens Axboe if (!timeo) { 8229c55e01cSJens Axboe ret = -EAGAIN; 8239c55e01cSJens Axboe break; 8249c55e01cSJens Axboe } 825ccf7abb9SEric Dumazet /* if __tcp_splice_read() got nothing while we have 826ccf7abb9SEric Dumazet * an skb in receive queue, we do not want to loop. 827ccf7abb9SEric Dumazet * This might happen with URG data. 828ccf7abb9SEric Dumazet */ 829ccf7abb9SEric Dumazet if (!skb_queue_empty(&sk->sk_receive_queue)) 830ccf7abb9SEric Dumazet break; 831dfbafc99SSabrina Dubroca sk_wait_data(sk, &timeo, NULL); 8329c55e01cSJens Axboe if (signal_pending(current)) { 8339c55e01cSJens Axboe ret = sock_intr_errno(timeo); 8349c55e01cSJens Axboe break; 8359c55e01cSJens Axboe } 8369c55e01cSJens Axboe continue; 8379c55e01cSJens Axboe } 8389c55e01cSJens Axboe tss.len -= ret; 8399c55e01cSJens Axboe spliced += ret; 8409c55e01cSJens Axboe 84133966dd0SWilly Tarreau if (!timeo) 84233966dd0SWilly Tarreau break; 8439c55e01cSJens Axboe release_sock(sk); 8449c55e01cSJens Axboe lock_sock(sk); 8459c55e01cSJens Axboe 8469c55e01cSJens Axboe if (sk->sk_err || sk->sk_state == TCP_CLOSE || 84733966dd0SWilly Tarreau (sk->sk_shutdown & RCV_SHUTDOWN) || 8489c55e01cSJens Axboe signal_pending(current)) 8499c55e01cSJens Axboe break; 8509c55e01cSJens Axboe } 8519c55e01cSJens Axboe 8529c55e01cSJens Axboe release_sock(sk); 8539c55e01cSJens Axboe 8549c55e01cSJens Axboe if (spliced) 8559c55e01cSJens Axboe return spliced; 8569c55e01cSJens Axboe 8579c55e01cSJens Axboe return ret; 8589c55e01cSJens Axboe } 8594bc2f18bSEric Dumazet EXPORT_SYMBOL(tcp_splice_read); 8609c55e01cSJens Axboe 861eb934478SEric Dumazet struct sk_buff *sk_stream_alloc_skb(struct sock *sk, int size, gfp_t gfp, 862eb934478SEric Dumazet bool force_schedule) 863f561d0f2SPavel Emelyanov { 864f561d0f2SPavel Emelyanov struct sk_buff *skb; 865f561d0f2SPavel Emelyanov 866f561d0f2SPavel Emelyanov /* The TCP header must be at least 32-bit aligned. */ 867f561d0f2SPavel Emelyanov size = ALIGN(size, 4); 868f561d0f2SPavel Emelyanov 8698e4d980aSEric Dumazet if (unlikely(tcp_under_memory_pressure(sk))) 8708e4d980aSEric Dumazet sk_mem_reclaim_partial(sk); 8718e4d980aSEric Dumazet 872f561d0f2SPavel Emelyanov skb = alloc_skb_fclone(size + sk->sk_prot->max_header, gfp); 8738e4d980aSEric Dumazet if (likely(skb)) { 874eb934478SEric Dumazet bool mem_scheduled; 8758e4d980aSEric Dumazet 876eb934478SEric Dumazet if (force_schedule) { 877eb934478SEric Dumazet mem_scheduled = true; 8788e4d980aSEric Dumazet sk_forced_mem_schedule(sk, skb->truesize); 8798e4d980aSEric Dumazet } else { 880eb934478SEric Dumazet mem_scheduled = sk_wmem_schedule(sk, skb->truesize); 8818e4d980aSEric Dumazet } 882eb934478SEric Dumazet if (likely(mem_scheduled)) { 883a21d4572SEric Dumazet skb_reserve(skb, sk->sk_prot->max_header); 884f561d0f2SPavel Emelyanov /* 885f561d0f2SPavel Emelyanov * Make sure that we have exactly size bytes 886f561d0f2SPavel Emelyanov * available to the caller, no more, no less. 887f561d0f2SPavel Emelyanov */ 88816fad69cSEric Dumazet skb->reserved_tailroom = skb->end - skb->tail - size; 889e2080072SEric Dumazet INIT_LIST_HEAD(&skb->tcp_tsorted_anchor); 890f561d0f2SPavel Emelyanov return skb; 891f561d0f2SPavel Emelyanov } 892f561d0f2SPavel Emelyanov __kfree_skb(skb); 893f561d0f2SPavel Emelyanov } else { 8945c52ba17SPavel Emelyanov sk->sk_prot->enter_memory_pressure(sk); 895f561d0f2SPavel Emelyanov sk_stream_moderate_sndbuf(sk); 896f561d0f2SPavel Emelyanov } 897f561d0f2SPavel Emelyanov return NULL; 898f561d0f2SPavel Emelyanov } 899f561d0f2SPavel Emelyanov 9000c54b85fSIlpo Järvinen static unsigned int tcp_xmit_size_goal(struct sock *sk, u32 mss_now, 9010c54b85fSIlpo Järvinen int large_allowed) 9020c54b85fSIlpo Järvinen { 9030c54b85fSIlpo Järvinen struct tcp_sock *tp = tcp_sk(sk); 9046c09fa09SEric Dumazet u32 new_size_goal, size_goal; 9050c54b85fSIlpo Järvinen 906605ad7f1SEric Dumazet if (!large_allowed || !sk_can_gso(sk)) 907605ad7f1SEric Dumazet return mss_now; 9080c54b85fSIlpo Järvinen 9096c09fa09SEric Dumazet /* Note : tcp_tso_autosize() will eventually split this later */ 9106c09fa09SEric Dumazet new_size_goal = sk->sk_gso_max_size - 1 - MAX_TCP_HEADER; 911605ad7f1SEric Dumazet new_size_goal = tcp_bound_to_half_wnd(tp, new_size_goal); 9122a3a041cSIlpo Järvinen 9132a3a041cSIlpo Järvinen /* We try hard to avoid divides here */ 914605ad7f1SEric Dumazet size_goal = tp->gso_segs * mss_now; 915605ad7f1SEric Dumazet if (unlikely(new_size_goal < size_goal || 916605ad7f1SEric Dumazet new_size_goal >= size_goal + mss_now)) { 917605ad7f1SEric Dumazet tp->gso_segs = min_t(u16, new_size_goal / mss_now, 9181485348dSBen Hutchings sk->sk_gso_max_segs); 919605ad7f1SEric Dumazet size_goal = tp->gso_segs * mss_now; 9200c54b85fSIlpo Järvinen } 9210c54b85fSIlpo Järvinen 922605ad7f1SEric Dumazet return max(size_goal, mss_now); 9230c54b85fSIlpo Järvinen } 9240c54b85fSIlpo Järvinen 9250c54b85fSIlpo Järvinen static int tcp_send_mss(struct sock *sk, int *size_goal, int flags) 9260c54b85fSIlpo Järvinen { 9270c54b85fSIlpo Järvinen int mss_now; 9280c54b85fSIlpo Järvinen 9290c54b85fSIlpo Järvinen mss_now = tcp_current_mss(sk); 9300c54b85fSIlpo Järvinen *size_goal = tcp_xmit_size_goal(sk, mss_now, !(flags & MSG_OOB)); 9310c54b85fSIlpo Järvinen 9320c54b85fSIlpo Järvinen return mss_now; 9330c54b85fSIlpo Järvinen } 9340c54b85fSIlpo Järvinen 935e3b5616aSDave Watson ssize_t do_tcp_sendpages(struct sock *sk, struct page *page, int offset, 93664022d0bSEric Dumazet size_t size, int flags) 9371da177e4SLinus Torvalds { 9381da177e4SLinus Torvalds struct tcp_sock *tp = tcp_sk(sk); 939c1b4a7e6SDavid S. Miller int mss_now, size_goal; 9401da177e4SLinus Torvalds int err; 9411da177e4SLinus Torvalds ssize_t copied; 9421da177e4SLinus Torvalds long timeo = sock_sndtimeo(sk, flags & MSG_DONTWAIT); 9431da177e4SLinus Torvalds 9448336886fSJerry Chu /* Wait for a connection to finish. One exception is TCP Fast Open 9458336886fSJerry Chu * (passive side) where data is allowed to be sent before a connection 9468336886fSJerry Chu * is fully established. 9478336886fSJerry Chu */ 9488336886fSJerry Chu if (((1 << sk->sk_state) & ~(TCPF_ESTABLISHED | TCPF_CLOSE_WAIT)) && 9498336886fSJerry Chu !tcp_passive_fastopen(sk)) { 950686a5624SYuvaraja Mariappan err = sk_stream_wait_connect(sk, &timeo); 951686a5624SYuvaraja Mariappan if (err != 0) 9521da177e4SLinus Torvalds goto out_err; 9538336886fSJerry Chu } 9541da177e4SLinus Torvalds 9559cd3e072SEric Dumazet sk_clear_bit(SOCKWQ_ASYNC_NOSPACE, sk); 9561da177e4SLinus Torvalds 9570c54b85fSIlpo Järvinen mss_now = tcp_send_mss(sk, &size_goal, flags); 9581da177e4SLinus Torvalds copied = 0; 9591da177e4SLinus Torvalds 9601da177e4SLinus Torvalds err = -EPIPE; 9611da177e4SLinus Torvalds if (sk->sk_err || (sk->sk_shutdown & SEND_SHUTDOWN)) 9620d6a775eSIlpo Järvinen goto out_err; 9631da177e4SLinus Torvalds 96464022d0bSEric Dumazet while (size > 0) { 965fe067e8aSDavid S. Miller struct sk_buff *skb = tcp_write_queue_tail(sk); 96638ba0a65SEric Dumazet int copy, i; 96738ba0a65SEric Dumazet bool can_coalesce; 9681da177e4SLinus Torvalds 96975c119afSEric Dumazet if (!skb || (copy = size_goal - skb->len) <= 0 || 970c134ecb8SMartin KaFai Lau !tcp_skb_can_collapse_to(skb)) { 9711da177e4SLinus Torvalds new_segment: 9721da177e4SLinus Torvalds if (!sk_stream_memory_free(sk)) 9731da177e4SLinus Torvalds goto wait_for_sndbuf; 9741da177e4SLinus Torvalds 975eb934478SEric Dumazet skb = sk_stream_alloc_skb(sk, 0, sk->sk_allocation, 97675c119afSEric Dumazet tcp_rtx_and_write_queues_empty(sk)); 9771da177e4SLinus Torvalds if (!skb) 9781da177e4SLinus Torvalds goto wait_for_memory; 9791da177e4SLinus Torvalds 9809e412ba7SIlpo Järvinen skb_entail(sk, skb); 981c1b4a7e6SDavid S. Miller copy = size_goal; 9821da177e4SLinus Torvalds } 9831da177e4SLinus Torvalds 9841da177e4SLinus Torvalds if (copy > size) 9851da177e4SLinus Torvalds copy = size; 9861da177e4SLinus Torvalds 9871da177e4SLinus Torvalds i = skb_shinfo(skb)->nr_frags; 9881da177e4SLinus Torvalds can_coalesce = skb_can_coalesce(skb, i, page, offset); 9895f74f82eSHans Westgaard Ry if (!can_coalesce && i >= sysctl_max_skb_frags) { 9901da177e4SLinus Torvalds tcp_mark_push(tp, skb); 9911da177e4SLinus Torvalds goto new_segment; 9921da177e4SLinus Torvalds } 9933ab224beSHideo Aoki if (!sk_wmem_schedule(sk, copy)) 9941da177e4SLinus Torvalds goto wait_for_memory; 9951da177e4SLinus Torvalds 9961da177e4SLinus Torvalds if (can_coalesce) { 9979e903e08SEric Dumazet skb_frag_size_add(&skb_shinfo(skb)->frags[i - 1], copy); 9981da177e4SLinus Torvalds } else { 9991da177e4SLinus Torvalds get_page(page); 10001da177e4SLinus Torvalds skb_fill_page_desc(skb, i, page, offset, copy); 10011da177e4SLinus Torvalds } 1002c9af6db4SPravin B Shelar skb_shinfo(skb)->tx_flags |= SKBTX_SHARED_FRAG; 1003cef401deSEric Dumazet 10041da177e4SLinus Torvalds skb->len += copy; 10051da177e4SLinus Torvalds skb->data_len += copy; 10061da177e4SLinus Torvalds skb->truesize += copy; 10071da177e4SLinus Torvalds sk->sk_wmem_queued += copy; 10083ab224beSHideo Aoki sk_mem_charge(sk, copy); 100984fa7933SPatrick McHardy skb->ip_summed = CHECKSUM_PARTIAL; 10101da177e4SLinus Torvalds tp->write_seq += copy; 10111da177e4SLinus Torvalds TCP_SKB_CB(skb)->end_seq += copy; 1012cd7d8498SEric Dumazet tcp_skb_pcount_set(skb, 0); 10131da177e4SLinus Torvalds 10141da177e4SLinus Torvalds if (!copied) 10154de075e0SEric Dumazet TCP_SKB_CB(skb)->tcp_flags &= ~TCPHDR_PSH; 10161da177e4SLinus Torvalds 10171da177e4SLinus Torvalds copied += copy; 101864022d0bSEric Dumazet offset += copy; 1019686a5624SYuvaraja Mariappan size -= copy; 1020ad02c4f5SSoheil Hassas Yeganeh if (!size) 10211da177e4SLinus Torvalds goto out; 10221da177e4SLinus Torvalds 102369d15067SHerbert Xu if (skb->len < size_goal || (flags & MSG_OOB)) 10241da177e4SLinus Torvalds continue; 10251da177e4SLinus Torvalds 10261da177e4SLinus Torvalds if (forced_push(tp)) { 10271da177e4SLinus Torvalds tcp_mark_push(tp, skb); 10289e412ba7SIlpo Järvinen __tcp_push_pending_frames(sk, mss_now, TCP_NAGLE_PUSH); 1029fe067e8aSDavid S. Miller } else if (skb == tcp_send_head(sk)) 10301da177e4SLinus Torvalds tcp_push_one(sk, mss_now); 10311da177e4SLinus Torvalds continue; 10321da177e4SLinus Torvalds 10331da177e4SLinus Torvalds wait_for_sndbuf: 10341da177e4SLinus Torvalds set_bit(SOCK_NOSPACE, &sk->sk_socket->flags); 10351da177e4SLinus Torvalds wait_for_memory: 1036f54b3111SEric Dumazet tcp_push(sk, flags & ~MSG_MORE, mss_now, 1037f54b3111SEric Dumazet TCP_NAGLE_PUSH, size_goal); 10381da177e4SLinus Torvalds 1039686a5624SYuvaraja Mariappan err = sk_stream_wait_memory(sk, &timeo); 1040686a5624SYuvaraja Mariappan if (err != 0) 10411da177e4SLinus Torvalds goto do_error; 10421da177e4SLinus Torvalds 10430c54b85fSIlpo Järvinen mss_now = tcp_send_mss(sk, &size_goal, flags); 10441da177e4SLinus Torvalds } 10451da177e4SLinus Torvalds 10461da177e4SLinus Torvalds out: 1047ad02c4f5SSoheil Hassas Yeganeh if (copied) { 10484e8cc228SEric Dumazet tcp_tx_timestamp(sk, sk->sk_tsflags); 1049ad02c4f5SSoheil Hassas Yeganeh if (!(flags & MSG_SENDPAGE_NOTLAST)) 1050f54b3111SEric Dumazet tcp_push(sk, flags, mss_now, tp->nonagle, size_goal); 1051ad02c4f5SSoheil Hassas Yeganeh } 10521da177e4SLinus Torvalds return copied; 10531da177e4SLinus Torvalds 10541da177e4SLinus Torvalds do_error: 10551da177e4SLinus Torvalds if (copied) 10561da177e4SLinus Torvalds goto out; 10571da177e4SLinus Torvalds out_err: 1058ce5ec440SJason Baron /* make sure we wake any epoll edge trigger waiter */ 1059b0f71bd3SFrancis Yan if (unlikely(skb_queue_len(&sk->sk_write_queue) == 0 && 1060b0f71bd3SFrancis Yan err == -EAGAIN)) { 1061ce5ec440SJason Baron sk->sk_write_space(sk); 1062b0f71bd3SFrancis Yan tcp_chrono_stop(sk, TCP_CHRONO_SNDBUF_LIMITED); 1063b0f71bd3SFrancis Yan } 10641da177e4SLinus Torvalds return sk_stream_error(sk, flags, err); 10651da177e4SLinus Torvalds } 1066e3b5616aSDave Watson EXPORT_SYMBOL_GPL(do_tcp_sendpages); 10671da177e4SLinus Torvalds 1068306b13ebSTom Herbert int tcp_sendpage_locked(struct sock *sk, struct page *page, int offset, 10691da177e4SLinus Torvalds size_t size, int flags) 10701da177e4SLinus Torvalds { 10711da177e4SLinus Torvalds if (!(sk->sk_route_caps & NETIF_F_SG) || 10729a49850dSTom Herbert !sk_check_csum_caps(sk)) 1073bd9dfc54SEric Dumazet return sock_no_sendpage_locked(sk, page, offset, size, flags); 10741da177e4SLinus Torvalds 1075d7722e85SSoheil Hassas Yeganeh tcp_rate_check_app_limited(sk); /* is sending application-limited? */ 1076d7722e85SSoheil Hassas Yeganeh 1077306b13ebSTom Herbert return do_tcp_sendpages(sk, page, offset, size, flags); 1078306b13ebSTom Herbert } 1079774c4673SDavid S. Miller EXPORT_SYMBOL_GPL(tcp_sendpage_locked); 1080306b13ebSTom Herbert 1081306b13ebSTom Herbert int tcp_sendpage(struct sock *sk, struct page *page, int offset, 1082306b13ebSTom Herbert size_t size, int flags) 1083306b13ebSTom Herbert { 1084306b13ebSTom Herbert int ret; 1085306b13ebSTom Herbert 1086306b13ebSTom Herbert lock_sock(sk); 1087306b13ebSTom Herbert ret = tcp_sendpage_locked(sk, page, offset, size, flags); 10881da177e4SLinus Torvalds release_sock(sk); 1089306b13ebSTom Herbert 1090306b13ebSTom Herbert return ret; 10911da177e4SLinus Torvalds } 10924bc2f18bSEric Dumazet EXPORT_SYMBOL(tcp_sendpage); 10931da177e4SLinus Torvalds 10943613b3dbSEric Dumazet /* Do not bother using a page frag for very small frames. 10953613b3dbSEric Dumazet * But use this heuristic only for the first skb in write queue. 10963613b3dbSEric Dumazet * 10973613b3dbSEric Dumazet * Having no payload in skb->head allows better SACK shifting 10983613b3dbSEric Dumazet * in tcp_shift_skb_data(), reducing sack/rack overhead, because 10993613b3dbSEric Dumazet * write queue has less skbs. 11003613b3dbSEric Dumazet * Each skb can hold up to MAX_SKB_FRAGS * 32Kbytes, or ~0.5 MB. 11013613b3dbSEric Dumazet * This also speeds up tso_fragment(), since it wont fallback 11023613b3dbSEric Dumazet * to tcp_fragment(). 11033613b3dbSEric Dumazet */ 11043613b3dbSEric Dumazet static int linear_payload_sz(bool first_skb) 11053613b3dbSEric Dumazet { 11063613b3dbSEric Dumazet if (first_skb) 11073613b3dbSEric Dumazet return SKB_WITH_OVERHEAD(2048 - MAX_TCP_HEADER); 11083613b3dbSEric Dumazet return 0; 11093613b3dbSEric Dumazet } 11103613b3dbSEric Dumazet 11113613b3dbSEric Dumazet static int select_size(const struct sock *sk, bool sg, bool first_skb) 11121da177e4SLinus Torvalds { 1113cf533ea5SEric Dumazet const struct tcp_sock *tp = tcp_sk(sk); 1114c1b4a7e6SDavid S. Miller int tmp = tp->mss_cache; 11151da177e4SLinus Torvalds 1116def87cf4SKrishna Kumar if (sg) { 1117f07d960dSEric Dumazet if (sk_can_gso(sk)) { 11183613b3dbSEric Dumazet tmp = linear_payload_sz(first_skb); 1119f07d960dSEric Dumazet } else { 1120b4e26f5eSDavid S. Miller int pgbreak = SKB_MAX_HEAD(MAX_TCP_HEADER); 1121b4e26f5eSDavid S. Miller 1122b4e26f5eSDavid S. Miller if (tmp >= pgbreak && 1123b4e26f5eSDavid S. Miller tmp <= pgbreak + (MAX_SKB_FRAGS - 1) * PAGE_SIZE) 1124b4e26f5eSDavid S. Miller tmp = pgbreak; 1125b4e26f5eSDavid S. Miller } 1126b4e26f5eSDavid S. Miller } 11271da177e4SLinus Torvalds 11281da177e4SLinus Torvalds return tmp; 11291da177e4SLinus Torvalds } 11301da177e4SLinus Torvalds 1131cf60af03SYuchung Cheng void tcp_free_fastopen_req(struct tcp_sock *tp) 1132cf60af03SYuchung Cheng { 113300db4124SIan Morris if (tp->fastopen_req) { 1134cf60af03SYuchung Cheng kfree(tp->fastopen_req); 1135cf60af03SYuchung Cheng tp->fastopen_req = NULL; 1136cf60af03SYuchung Cheng } 1137cf60af03SYuchung Cheng } 1138cf60af03SYuchung Cheng 1139f5ddcbbbSEric Dumazet static int tcp_sendmsg_fastopen(struct sock *sk, struct msghdr *msg, 1140f5ddcbbbSEric Dumazet int *copied, size_t size) 1141cf60af03SYuchung Cheng { 1142cf60af03SYuchung Cheng struct tcp_sock *tp = tcp_sk(sk); 114319f6d3f3SWei Wang struct inet_sock *inet = inet_sk(sk); 1144ba615f67SWei Wang struct sockaddr *uaddr = msg->msg_name; 1145cf60af03SYuchung Cheng int err, flags; 1146cf60af03SYuchung Cheng 1147e1cfcbe8SHaishuang Yan if (!(sock_net(sk)->ipv4.sysctl_tcp_fastopen & TFO_CLIENT_ENABLE) || 1148ba615f67SWei Wang (uaddr && msg->msg_namelen >= sizeof(uaddr->sa_family) && 1149ba615f67SWei Wang uaddr->sa_family == AF_UNSPEC)) 1150cf60af03SYuchung Cheng return -EOPNOTSUPP; 115100db4124SIan Morris if (tp->fastopen_req) 1152cf60af03SYuchung Cheng return -EALREADY; /* Another Fast Open is in progress */ 1153cf60af03SYuchung Cheng 1154cf60af03SYuchung Cheng tp->fastopen_req = kzalloc(sizeof(struct tcp_fastopen_request), 1155cf60af03SYuchung Cheng sk->sk_allocation); 115651456b29SIan Morris if (unlikely(!tp->fastopen_req)) 1157cf60af03SYuchung Cheng return -ENOBUFS; 1158cf60af03SYuchung Cheng tp->fastopen_req->data = msg; 1159f5ddcbbbSEric Dumazet tp->fastopen_req->size = size; 1160cf60af03SYuchung Cheng 116119f6d3f3SWei Wang if (inet->defer_connect) { 116219f6d3f3SWei Wang err = tcp_connect(sk); 116319f6d3f3SWei Wang /* Same failure procedure as in tcp_v4/6_connect */ 116419f6d3f3SWei Wang if (err) { 116519f6d3f3SWei Wang tcp_set_state(sk, TCP_CLOSE); 116619f6d3f3SWei Wang inet->inet_dport = 0; 116719f6d3f3SWei Wang sk->sk_route_caps = 0; 116819f6d3f3SWei Wang } 116919f6d3f3SWei Wang } 1170cf60af03SYuchung Cheng flags = (msg->msg_flags & MSG_DONTWAIT) ? O_NONBLOCK : 0; 1171ba615f67SWei Wang err = __inet_stream_connect(sk->sk_socket, uaddr, 11723979ad7eSWilly Tarreau msg->msg_namelen, flags, 1); 11737db92362SWei Wang /* fastopen_req could already be freed in __inet_stream_connect 11747db92362SWei Wang * if the connection times out or gets rst 11757db92362SWei Wang */ 11767db92362SWei Wang if (tp->fastopen_req) { 1177f5ddcbbbSEric Dumazet *copied = tp->fastopen_req->copied; 1178cf60af03SYuchung Cheng tcp_free_fastopen_req(tp); 11797db92362SWei Wang inet->defer_connect = 0; 11807db92362SWei Wang } 1181cf60af03SYuchung Cheng return err; 1182cf60af03SYuchung Cheng } 1183cf60af03SYuchung Cheng 1184306b13ebSTom Herbert int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) 11851da177e4SLinus Torvalds { 11861da177e4SLinus Torvalds struct tcp_sock *tp = tcp_sk(sk); 1187f214f915SWillem de Bruijn struct ubuf_info *uarg = NULL; 11881da177e4SLinus Torvalds struct sk_buff *skb; 1189c14ac945SSoheil Hassas Yeganeh struct sockcm_cookie sockc; 119057be5bdaSAl Viro int flags, err, copied = 0; 119157be5bdaSAl Viro int mss_now = 0, size_goal, copied_syn = 0; 1192d4011239SEric Dumazet bool process_backlog = false; 1193690e99c4SEric Dumazet bool sg; 11941da177e4SLinus Torvalds long timeo; 11951da177e4SLinus Torvalds 11961da177e4SLinus Torvalds flags = msg->msg_flags; 1197f214f915SWillem de Bruijn 1198f214f915SWillem de Bruijn if (flags & MSG_ZEROCOPY && size) { 1199f214f915SWillem de Bruijn if (sk->sk_state != TCP_ESTABLISHED) { 1200f214f915SWillem de Bruijn err = -EINVAL; 1201f214f915SWillem de Bruijn goto out_err; 1202f214f915SWillem de Bruijn } 1203f214f915SWillem de Bruijn 120475c119afSEric Dumazet skb = tcp_write_queue_tail(sk); 1205f214f915SWillem de Bruijn uarg = sock_zerocopy_realloc(sk, size, skb_zcopy(skb)); 1206f214f915SWillem de Bruijn if (!uarg) { 1207f214f915SWillem de Bruijn err = -ENOBUFS; 1208f214f915SWillem de Bruijn goto out_err; 1209f214f915SWillem de Bruijn } 1210f214f915SWillem de Bruijn 1211f214f915SWillem de Bruijn if (!(sk_check_csum_caps(sk) && sk->sk_route_caps & NETIF_F_SG)) 1212f214f915SWillem de Bruijn uarg->zerocopy = 0; 1213f214f915SWillem de Bruijn } 1214f214f915SWillem de Bruijn 121519f6d3f3SWei Wang if (unlikely(flags & MSG_FASTOPEN || inet_sk(sk)->defer_connect)) { 1216f5ddcbbbSEric Dumazet err = tcp_sendmsg_fastopen(sk, msg, &copied_syn, size); 1217cf60af03SYuchung Cheng if (err == -EINPROGRESS && copied_syn > 0) 1218cf60af03SYuchung Cheng goto out; 1219cf60af03SYuchung Cheng else if (err) 1220cf60af03SYuchung Cheng goto out_err; 1221cf60af03SYuchung Cheng } 1222cf60af03SYuchung Cheng 12231da177e4SLinus Torvalds timeo = sock_sndtimeo(sk, flags & MSG_DONTWAIT); 12241da177e4SLinus Torvalds 1225d7722e85SSoheil Hassas Yeganeh tcp_rate_check_app_limited(sk); /* is sending application-limited? */ 1226d7722e85SSoheil Hassas Yeganeh 12278336886fSJerry Chu /* Wait for a connection to finish. One exception is TCP Fast Open 12288336886fSJerry Chu * (passive side) where data is allowed to be sent before a connection 12298336886fSJerry Chu * is fully established. 12308336886fSJerry Chu */ 12318336886fSJerry Chu if (((1 << sk->sk_state) & ~(TCPF_ESTABLISHED | TCPF_CLOSE_WAIT)) && 12328336886fSJerry Chu !tcp_passive_fastopen(sk)) { 1233686a5624SYuvaraja Mariappan err = sk_stream_wait_connect(sk, &timeo); 1234686a5624SYuvaraja Mariappan if (err != 0) 1235cf60af03SYuchung Cheng goto do_error; 12368336886fSJerry Chu } 12371da177e4SLinus Torvalds 1238c0e88ff0SPavel Emelyanov if (unlikely(tp->repair)) { 1239c0e88ff0SPavel Emelyanov if (tp->repair_queue == TCP_RECV_QUEUE) { 1240c0e88ff0SPavel Emelyanov copied = tcp_send_rcvq(sk, msg, size); 12415924f17aSChristoph Paasch goto out_nopush; 1242c0e88ff0SPavel Emelyanov } 1243c0e88ff0SPavel Emelyanov 1244c0e88ff0SPavel Emelyanov err = -EINVAL; 1245c0e88ff0SPavel Emelyanov if (tp->repair_queue == TCP_NO_QUEUE) 1246c0e88ff0SPavel Emelyanov goto out_err; 1247c0e88ff0SPavel Emelyanov 1248c0e88ff0SPavel Emelyanov /* 'common' sending to sendq */ 1249c0e88ff0SPavel Emelyanov } 1250c0e88ff0SPavel Emelyanov 1251c14ac945SSoheil Hassas Yeganeh sockc.tsflags = sk->sk_tsflags; 1252c14ac945SSoheil Hassas Yeganeh if (msg->msg_controllen) { 1253c14ac945SSoheil Hassas Yeganeh err = sock_cmsg_send(sk, msg, &sockc); 1254c14ac945SSoheil Hassas Yeganeh if (unlikely(err)) { 1255c14ac945SSoheil Hassas Yeganeh err = -EINVAL; 1256c14ac945SSoheil Hassas Yeganeh goto out_err; 1257c14ac945SSoheil Hassas Yeganeh } 1258c14ac945SSoheil Hassas Yeganeh } 1259c14ac945SSoheil Hassas Yeganeh 12601da177e4SLinus Torvalds /* This should be in poll */ 12619cd3e072SEric Dumazet sk_clear_bit(SOCKWQ_ASYNC_NOSPACE, sk); 12621da177e4SLinus Torvalds 12631da177e4SLinus Torvalds /* Ok commence sending. */ 12641da177e4SLinus Torvalds copied = 0; 12651da177e4SLinus Torvalds 1266d41a69f1SEric Dumazet restart: 1267d41a69f1SEric Dumazet mss_now = tcp_send_mss(sk, &size_goal, flags); 1268d41a69f1SEric Dumazet 12691da177e4SLinus Torvalds err = -EPIPE; 12701da177e4SLinus Torvalds if (sk->sk_err || (sk->sk_shutdown & SEND_SHUTDOWN)) 127179d8665bSEric Dumazet goto do_error; 12721da177e4SLinus Torvalds 1273690e99c4SEric Dumazet sg = !!(sk->sk_route_caps & NETIF_F_SG); 1274def87cf4SKrishna Kumar 127501e97e65SAl Viro while (msg_data_left(msg)) { 12766828b92bSHerbert Xu int copy = 0; 12776828b92bSHerbert Xu int max = size_goal; 12781da177e4SLinus Torvalds 1279fe067e8aSDavid S. Miller skb = tcp_write_queue_tail(sk); 128075c119afSEric Dumazet if (skb) { 12816828b92bSHerbert Xu if (skb->ip_summed == CHECKSUM_NONE) 12826828b92bSHerbert Xu max = mss_now; 12836828b92bSHerbert Xu copy = max - skb->len; 12846828b92bSHerbert Xu } 12851da177e4SLinus Torvalds 1286c134ecb8SMartin KaFai Lau if (copy <= 0 || !tcp_skb_can_collapse_to(skb)) { 12873613b3dbSEric Dumazet bool first_skb; 12883613b3dbSEric Dumazet 12891da177e4SLinus Torvalds new_segment: 12901da177e4SLinus Torvalds /* Allocate new segment. If the interface is SG, 12911da177e4SLinus Torvalds * allocate skb fitting to single page. 12921da177e4SLinus Torvalds */ 12931da177e4SLinus Torvalds if (!sk_stream_memory_free(sk)) 12941da177e4SLinus Torvalds goto wait_for_sndbuf; 12951da177e4SLinus Torvalds 1296d4011239SEric Dumazet if (process_backlog && sk_flush_backlog(sk)) { 1297d4011239SEric Dumazet process_backlog = false; 1298d41a69f1SEric Dumazet goto restart; 1299d4011239SEric Dumazet } 130075c119afSEric Dumazet first_skb = tcp_rtx_and_write_queues_empty(sk); 1301def87cf4SKrishna Kumar skb = sk_stream_alloc_skb(sk, 13023613b3dbSEric Dumazet select_size(sk, sg, first_skb), 1303eb934478SEric Dumazet sk->sk_allocation, 13043613b3dbSEric Dumazet first_skb); 13051da177e4SLinus Torvalds if (!skb) 13061da177e4SLinus Torvalds goto wait_for_memory; 13071da177e4SLinus Torvalds 1308d4011239SEric Dumazet process_backlog = true; 13091da177e4SLinus Torvalds /* 13101da177e4SLinus Torvalds * Check whether we can use HW checksum. 13111da177e4SLinus Torvalds */ 13129a49850dSTom Herbert if (sk_check_csum_caps(sk)) 131384fa7933SPatrick McHardy skb->ip_summed = CHECKSUM_PARTIAL; 13141da177e4SLinus Torvalds 13159e412ba7SIlpo Järvinen skb_entail(sk, skb); 1316c1b4a7e6SDavid S. Miller copy = size_goal; 13176828b92bSHerbert Xu max = size_goal; 13189d186cacSAndrey Vagin 13199d186cacSAndrey Vagin /* All packets are restored as if they have 13209d186cacSAndrey Vagin * already been sent. skb_mstamp isn't set to 13219d186cacSAndrey Vagin * avoid wrong rtt estimation. 13229d186cacSAndrey Vagin */ 13239d186cacSAndrey Vagin if (tp->repair) 13249d186cacSAndrey Vagin TCP_SKB_CB(skb)->sacked |= TCPCB_REPAIRED; 13251da177e4SLinus Torvalds } 13261da177e4SLinus Torvalds 13271da177e4SLinus Torvalds /* Try to append data to the end of skb. */ 132801e97e65SAl Viro if (copy > msg_data_left(msg)) 132901e97e65SAl Viro copy = msg_data_left(msg); 13301da177e4SLinus Torvalds 13311da177e4SLinus Torvalds /* Where to copy to? */ 1332a21d4572SEric Dumazet if (skb_availroom(skb) > 0) { 13331da177e4SLinus Torvalds /* We have some space in skb head. Superb! */ 1334a21d4572SEric Dumazet copy = min_t(int, copy, skb_availroom(skb)); 133557be5bdaSAl Viro err = skb_add_data_nocache(sk, skb, &msg->msg_iter, copy); 1336c6e1a0d1STom Herbert if (err) 13371da177e4SLinus Torvalds goto do_fault; 1338f214f915SWillem de Bruijn } else if (!uarg || !uarg->zerocopy) { 13395640f768SEric Dumazet bool merge = true; 13401da177e4SLinus Torvalds int i = skb_shinfo(skb)->nr_frags; 13415640f768SEric Dumazet struct page_frag *pfrag = sk_page_frag(sk); 1342761965eaSEric Dumazet 13435640f768SEric Dumazet if (!sk_page_frag_refill(sk, pfrag)) 13445640f768SEric Dumazet goto wait_for_memory; 1345761965eaSEric Dumazet 13465640f768SEric Dumazet if (!skb_can_coalesce(skb, i, pfrag->page, 13475640f768SEric Dumazet pfrag->offset)) { 1348ac9e70b1SEric Dumazet if (i >= sysctl_max_skb_frags || !sg) { 13491da177e4SLinus Torvalds tcp_mark_push(tp, skb); 13501da177e4SLinus Torvalds goto new_segment; 13511da177e4SLinus Torvalds } 13525640f768SEric Dumazet merge = false; 13535640f768SEric Dumazet } 1354ef015786SHerbert Xu 13555640f768SEric Dumazet copy = min_t(int, copy, pfrag->size - pfrag->offset); 1356ef015786SHerbert Xu 13573ab224beSHideo Aoki if (!sk_wmem_schedule(sk, copy)) 1358ef015786SHerbert Xu goto wait_for_memory; 13591da177e4SLinus Torvalds 136057be5bdaSAl Viro err = skb_copy_to_page_nocache(sk, &msg->msg_iter, skb, 13615640f768SEric Dumazet pfrag->page, 13625640f768SEric Dumazet pfrag->offset, 13635640f768SEric Dumazet copy); 13645640f768SEric Dumazet if (err) 13651da177e4SLinus Torvalds goto do_error; 13661da177e4SLinus Torvalds 13671da177e4SLinus Torvalds /* Update the skb. */ 13681da177e4SLinus Torvalds if (merge) { 13699e903e08SEric Dumazet skb_frag_size_add(&skb_shinfo(skb)->frags[i - 1], copy); 13701da177e4SLinus Torvalds } else { 13715640f768SEric Dumazet skb_fill_page_desc(skb, i, pfrag->page, 13725640f768SEric Dumazet pfrag->offset, copy); 13734e33e346SEric Dumazet page_ref_inc(pfrag->page); 13741da177e4SLinus Torvalds } 13755640f768SEric Dumazet pfrag->offset += copy; 1376f214f915SWillem de Bruijn } else { 1377f214f915SWillem de Bruijn err = skb_zerocopy_iter_stream(sk, skb, msg, copy, uarg); 1378f214f915SWillem de Bruijn if (err == -EMSGSIZE || err == -EEXIST) 1379f214f915SWillem de Bruijn goto new_segment; 1380f214f915SWillem de Bruijn if (err < 0) 1381f214f915SWillem de Bruijn goto do_error; 1382f214f915SWillem de Bruijn copy = err; 13831da177e4SLinus Torvalds } 13841da177e4SLinus Torvalds 13851da177e4SLinus Torvalds if (!copied) 13864de075e0SEric Dumazet TCP_SKB_CB(skb)->tcp_flags &= ~TCPHDR_PSH; 13871da177e4SLinus Torvalds 13881da177e4SLinus Torvalds tp->write_seq += copy; 13891da177e4SLinus Torvalds TCP_SKB_CB(skb)->end_seq += copy; 1390cd7d8498SEric Dumazet tcp_skb_pcount_set(skb, 0); 13911da177e4SLinus Torvalds 13921da177e4SLinus Torvalds copied += copy; 139301e97e65SAl Viro if (!msg_data_left(msg)) { 1394c134ecb8SMartin KaFai Lau if (unlikely(flags & MSG_EOR)) 1395c134ecb8SMartin KaFai Lau TCP_SKB_CB(skb)->eor = 1; 13961da177e4SLinus Torvalds goto out; 13974ed2d765SWillem de Bruijn } 13981da177e4SLinus Torvalds 1399c0e88ff0SPavel Emelyanov if (skb->len < max || (flags & MSG_OOB) || unlikely(tp->repair)) 14001da177e4SLinus Torvalds continue; 14011da177e4SLinus Torvalds 14021da177e4SLinus Torvalds if (forced_push(tp)) { 14031da177e4SLinus Torvalds tcp_mark_push(tp, skb); 14049e412ba7SIlpo Järvinen __tcp_push_pending_frames(sk, mss_now, TCP_NAGLE_PUSH); 1405fe067e8aSDavid S. Miller } else if (skb == tcp_send_head(sk)) 14061da177e4SLinus Torvalds tcp_push_one(sk, mss_now); 14071da177e4SLinus Torvalds continue; 14081da177e4SLinus Torvalds 14091da177e4SLinus Torvalds wait_for_sndbuf: 14101da177e4SLinus Torvalds set_bit(SOCK_NOSPACE, &sk->sk_socket->flags); 14111da177e4SLinus Torvalds wait_for_memory: 1412ec342325SAndrew Vagin if (copied) 1413f54b3111SEric Dumazet tcp_push(sk, flags & ~MSG_MORE, mss_now, 1414f54b3111SEric Dumazet TCP_NAGLE_PUSH, size_goal); 14151da177e4SLinus Torvalds 1416686a5624SYuvaraja Mariappan err = sk_stream_wait_memory(sk, &timeo); 1417686a5624SYuvaraja Mariappan if (err != 0) 14181da177e4SLinus Torvalds goto do_error; 14191da177e4SLinus Torvalds 14200c54b85fSIlpo Järvinen mss_now = tcp_send_mss(sk, &size_goal, flags); 14211da177e4SLinus Torvalds } 14221da177e4SLinus Torvalds 14231da177e4SLinus Torvalds out: 1424ad02c4f5SSoheil Hassas Yeganeh if (copied) { 14254e8cc228SEric Dumazet tcp_tx_timestamp(sk, sockc.tsflags); 1426f54b3111SEric Dumazet tcp_push(sk, flags, mss_now, tp->nonagle, size_goal); 1427ad02c4f5SSoheil Hassas Yeganeh } 14285924f17aSChristoph Paasch out_nopush: 1429f214f915SWillem de Bruijn sock_zerocopy_put(uarg); 1430cf60af03SYuchung Cheng return copied + copied_syn; 14311da177e4SLinus Torvalds 14321da177e4SLinus Torvalds do_fault: 14331da177e4SLinus Torvalds if (!skb->len) { 1434fe067e8aSDavid S. Miller tcp_unlink_write_queue(skb, sk); 1435fe067e8aSDavid S. Miller /* It is the one place in all of TCP, except connection 1436fe067e8aSDavid S. Miller * reset, where we can be unlinking the send_head. 1437fe067e8aSDavid S. Miller */ 1438fe067e8aSDavid S. Miller tcp_check_send_head(sk, skb); 14393ab224beSHideo Aoki sk_wmem_free_skb(sk, skb); 14401da177e4SLinus Torvalds } 14411da177e4SLinus Torvalds 14421da177e4SLinus Torvalds do_error: 1443cf60af03SYuchung Cheng if (copied + copied_syn) 14441da177e4SLinus Torvalds goto out; 14451da177e4SLinus Torvalds out_err: 1446f214f915SWillem de Bruijn sock_zerocopy_put_abort(uarg); 14471da177e4SLinus Torvalds err = sk_stream_error(sk, flags, err); 1448ce5ec440SJason Baron /* make sure we wake any epoll edge trigger waiter */ 1449b0f71bd3SFrancis Yan if (unlikely(skb_queue_len(&sk->sk_write_queue) == 0 && 1450b0f71bd3SFrancis Yan err == -EAGAIN)) { 1451ce5ec440SJason Baron sk->sk_write_space(sk); 1452b0f71bd3SFrancis Yan tcp_chrono_stop(sk, TCP_CHRONO_SNDBUF_LIMITED); 1453b0f71bd3SFrancis Yan } 14541da177e4SLinus Torvalds return err; 14551da177e4SLinus Torvalds } 1456774c4673SDavid S. Miller EXPORT_SYMBOL_GPL(tcp_sendmsg_locked); 1457306b13ebSTom Herbert 1458306b13ebSTom Herbert int tcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) 1459306b13ebSTom Herbert { 1460306b13ebSTom Herbert int ret; 1461306b13ebSTom Herbert 1462306b13ebSTom Herbert lock_sock(sk); 1463306b13ebSTom Herbert ret = tcp_sendmsg_locked(sk, msg, size); 1464306b13ebSTom Herbert release_sock(sk); 1465306b13ebSTom Herbert 1466306b13ebSTom Herbert return ret; 1467306b13ebSTom Herbert } 14684bc2f18bSEric Dumazet EXPORT_SYMBOL(tcp_sendmsg); 14691da177e4SLinus Torvalds 14701da177e4SLinus Torvalds /* 14711da177e4SLinus Torvalds * Handle reading urgent data. BSD has very simple semantics for 14721da177e4SLinus Torvalds * this, no blocking and very strange errors 8) 14731da177e4SLinus Torvalds */ 14741da177e4SLinus Torvalds 1475377f0a08SRami Rosen static int tcp_recv_urg(struct sock *sk, struct msghdr *msg, int len, int flags) 14761da177e4SLinus Torvalds { 14771da177e4SLinus Torvalds struct tcp_sock *tp = tcp_sk(sk); 14781da177e4SLinus Torvalds 14791da177e4SLinus Torvalds /* No URG data to read. */ 14801da177e4SLinus Torvalds if (sock_flag(sk, SOCK_URGINLINE) || !tp->urg_data || 14811da177e4SLinus Torvalds tp->urg_data == TCP_URG_READ) 14821da177e4SLinus Torvalds return -EINVAL; /* Yes this is right ! */ 14831da177e4SLinus Torvalds 14841da177e4SLinus Torvalds if (sk->sk_state == TCP_CLOSE && !sock_flag(sk, SOCK_DONE)) 14851da177e4SLinus Torvalds return -ENOTCONN; 14861da177e4SLinus Torvalds 14871da177e4SLinus Torvalds if (tp->urg_data & TCP_URG_VALID) { 14881da177e4SLinus Torvalds int err = 0; 14891da177e4SLinus Torvalds char c = tp->urg_data; 14901da177e4SLinus Torvalds 14911da177e4SLinus Torvalds if (!(flags & MSG_PEEK)) 14921da177e4SLinus Torvalds tp->urg_data = TCP_URG_READ; 14931da177e4SLinus Torvalds 14941da177e4SLinus Torvalds /* Read urgent data. */ 14951da177e4SLinus Torvalds msg->msg_flags |= MSG_OOB; 14961da177e4SLinus Torvalds 14971da177e4SLinus Torvalds if (len > 0) { 14981da177e4SLinus Torvalds if (!(flags & MSG_TRUNC)) 14997eab8d9eSAl Viro err = memcpy_to_msg(msg, &c, 1); 15001da177e4SLinus Torvalds len = 1; 15011da177e4SLinus Torvalds } else 15021da177e4SLinus Torvalds msg->msg_flags |= MSG_TRUNC; 15031da177e4SLinus Torvalds 15041da177e4SLinus Torvalds return err ? -EFAULT : len; 15051da177e4SLinus Torvalds } 15061da177e4SLinus Torvalds 15071da177e4SLinus Torvalds if (sk->sk_state == TCP_CLOSE || (sk->sk_shutdown & RCV_SHUTDOWN)) 15081da177e4SLinus Torvalds return 0; 15091da177e4SLinus Torvalds 15101da177e4SLinus Torvalds /* Fixed the recv(..., MSG_OOB) behaviour. BSD docs and 15111da177e4SLinus Torvalds * the available implementations agree in this case: 15121da177e4SLinus Torvalds * this call should never block, independent of the 15131da177e4SLinus Torvalds * blocking state of the socket. 15141da177e4SLinus Torvalds * Mike <pall@rz.uni-karlsruhe.de> 15151da177e4SLinus Torvalds */ 15161da177e4SLinus Torvalds return -EAGAIN; 15171da177e4SLinus Torvalds } 15181da177e4SLinus Torvalds 1519c0e88ff0SPavel Emelyanov static int tcp_peek_sndq(struct sock *sk, struct msghdr *msg, int len) 1520c0e88ff0SPavel Emelyanov { 1521c0e88ff0SPavel Emelyanov struct sk_buff *skb; 1522c0e88ff0SPavel Emelyanov int copied = 0, err = 0; 1523c0e88ff0SPavel Emelyanov 1524c0e88ff0SPavel Emelyanov /* XXX -- need to support SO_PEEK_OFF */ 1525c0e88ff0SPavel Emelyanov 152675c119afSEric Dumazet skb_rbtree_walk(skb, &sk->tcp_rtx_queue) { 152775c119afSEric Dumazet err = skb_copy_datagram_msg(skb, 0, msg, skb->len); 152875c119afSEric Dumazet if (err) 152975c119afSEric Dumazet return err; 153075c119afSEric Dumazet copied += skb->len; 153175c119afSEric Dumazet } 153275c119afSEric Dumazet 1533c0e88ff0SPavel Emelyanov skb_queue_walk(&sk->sk_write_queue, skb) { 153451f3d02bSDavid S. Miller err = skb_copy_datagram_msg(skb, 0, msg, skb->len); 1535c0e88ff0SPavel Emelyanov if (err) 1536c0e88ff0SPavel Emelyanov break; 1537c0e88ff0SPavel Emelyanov 1538c0e88ff0SPavel Emelyanov copied += skb->len; 1539c0e88ff0SPavel Emelyanov } 1540c0e88ff0SPavel Emelyanov 1541c0e88ff0SPavel Emelyanov return err ?: copied; 1542c0e88ff0SPavel Emelyanov } 1543c0e88ff0SPavel Emelyanov 15441da177e4SLinus Torvalds /* Clean up the receive buffer for full frames taken by the user, 15451da177e4SLinus Torvalds * then send an ACK if necessary. COPIED is the number of bytes 15461da177e4SLinus Torvalds * tcp_recvmsg has given to the user so far, it speeds up the 15471da177e4SLinus Torvalds * calculation of whether or not we must ACK for the sake of 15481da177e4SLinus Torvalds * a window update. 15491da177e4SLinus Torvalds */ 15503f334078SDan Williams static void tcp_cleanup_rbuf(struct sock *sk, int copied) 15511da177e4SLinus Torvalds { 15521da177e4SLinus Torvalds struct tcp_sock *tp = tcp_sk(sk); 1553a2a385d6SEric Dumazet bool time_to_ack = false; 15541da177e4SLinus Torvalds 15551da177e4SLinus Torvalds struct sk_buff *skb = skb_peek(&sk->sk_receive_queue); 15561da177e4SLinus Torvalds 1557d792c100SIlpo Järvinen WARN(skb && !before(tp->copied_seq, TCP_SKB_CB(skb)->end_seq), 15582af6fd8bSJoe Perches "cleanup rbuf bug: copied %X seq %X rcvnxt %X\n", 1559d792c100SIlpo Järvinen tp->copied_seq, TCP_SKB_CB(skb)->end_seq, tp->rcv_nxt); 15601da177e4SLinus Torvalds 1561463c84b9SArnaldo Carvalho de Melo if (inet_csk_ack_scheduled(sk)) { 1562463c84b9SArnaldo Carvalho de Melo const struct inet_connection_sock *icsk = inet_csk(sk); 15631da177e4SLinus Torvalds /* Delayed ACKs frequently hit locked sockets during bulk 15641da177e4SLinus Torvalds * receive. */ 1565463c84b9SArnaldo Carvalho de Melo if (icsk->icsk_ack.blocked || 15661da177e4SLinus Torvalds /* Once-per-two-segments ACK was not sent by tcp_input.c */ 1567463c84b9SArnaldo Carvalho de Melo tp->rcv_nxt - tp->rcv_wup > icsk->icsk_ack.rcv_mss || 15681da177e4SLinus Torvalds /* 15691da177e4SLinus Torvalds * If this read emptied read buffer, we send ACK, if 15701da177e4SLinus Torvalds * connection is not bidirectional, user drained 15711da177e4SLinus Torvalds * receive buffer and there was a small segment 15721da177e4SLinus Torvalds * in queue. 15731da177e4SLinus Torvalds */ 15741ef9696cSAlexey Kuznetsov (copied > 0 && 15751ef9696cSAlexey Kuznetsov ((icsk->icsk_ack.pending & ICSK_ACK_PUSHED2) || 15761ef9696cSAlexey Kuznetsov ((icsk->icsk_ack.pending & ICSK_ACK_PUSHED) && 15771ef9696cSAlexey Kuznetsov !icsk->icsk_ack.pingpong)) && 15781ef9696cSAlexey Kuznetsov !atomic_read(&sk->sk_rmem_alloc))) 1579a2a385d6SEric Dumazet time_to_ack = true; 15801da177e4SLinus Torvalds } 15811da177e4SLinus Torvalds 15821da177e4SLinus Torvalds /* We send an ACK if we can now advertise a non-zero window 15831da177e4SLinus Torvalds * which has been raised "significantly". 15841da177e4SLinus Torvalds * 15851da177e4SLinus Torvalds * Even if window raised up to infinity, do not send window open ACK 15861da177e4SLinus Torvalds * in states, where we will not receive more. It is useless. 15871da177e4SLinus Torvalds */ 15881da177e4SLinus Torvalds if (copied > 0 && !time_to_ack && !(sk->sk_shutdown & RCV_SHUTDOWN)) { 15891da177e4SLinus Torvalds __u32 rcv_window_now = tcp_receive_window(tp); 15901da177e4SLinus Torvalds 15911da177e4SLinus Torvalds /* Optimize, __tcp_select_window() is not cheap. */ 15921da177e4SLinus Torvalds if (2*rcv_window_now <= tp->window_clamp) { 15931da177e4SLinus Torvalds __u32 new_window = __tcp_select_window(sk); 15941da177e4SLinus Torvalds 15951da177e4SLinus Torvalds /* Send ACK now, if this read freed lots of space 15961da177e4SLinus Torvalds * in our buffer. Certainly, new_window is new window. 15971da177e4SLinus Torvalds * We can advertise it now, if it is not less than current one. 15981da177e4SLinus Torvalds * "Lots" means "at least twice" here. 15991da177e4SLinus Torvalds */ 16001da177e4SLinus Torvalds if (new_window && new_window >= 2 * rcv_window_now) 1601a2a385d6SEric Dumazet time_to_ack = true; 16021da177e4SLinus Torvalds } 16031da177e4SLinus Torvalds } 16041da177e4SLinus Torvalds if (time_to_ack) 16051da177e4SLinus Torvalds tcp_send_ack(sk); 16061da177e4SLinus Torvalds } 16071da177e4SLinus Torvalds 1608f26845b4SEric Dumazet static struct sk_buff *tcp_recv_skb(struct sock *sk, u32 seq, u32 *off) 16091da177e4SLinus Torvalds { 16101da177e4SLinus Torvalds struct sk_buff *skb; 16111da177e4SLinus Torvalds u32 offset; 16121da177e4SLinus Torvalds 1613f26845b4SEric Dumazet while ((skb = skb_peek(&sk->sk_receive_queue)) != NULL) { 16141da177e4SLinus Torvalds offset = seq - TCP_SKB_CB(skb)->seq; 16159d691539SEric Dumazet if (unlikely(TCP_SKB_CB(skb)->tcp_flags & TCPHDR_SYN)) { 16169d691539SEric Dumazet pr_err_once("%s: found a SYN, please report !\n", __func__); 16171da177e4SLinus Torvalds offset--; 16189d691539SEric Dumazet } 1619e11ecddfSEric Dumazet if (offset < skb->len || (TCP_SKB_CB(skb)->tcp_flags & TCPHDR_FIN)) { 16201da177e4SLinus Torvalds *off = offset; 16211da177e4SLinus Torvalds return skb; 16221da177e4SLinus Torvalds } 1623f26845b4SEric Dumazet /* This looks weird, but this can happen if TCP collapsing 1624f26845b4SEric Dumazet * splitted a fat GRO packet, while we released socket lock 1625f26845b4SEric Dumazet * in skb_splice_bits() 1626f26845b4SEric Dumazet */ 16277bced397SDan Williams sk_eat_skb(sk, skb); 16281da177e4SLinus Torvalds } 16291da177e4SLinus Torvalds return NULL; 16301da177e4SLinus Torvalds } 16311da177e4SLinus Torvalds 16321da177e4SLinus Torvalds /* 16331da177e4SLinus Torvalds * This routine provides an alternative to tcp_recvmsg() for routines 16341da177e4SLinus Torvalds * that would like to handle copying from skbuffs directly in 'sendfile' 16351da177e4SLinus Torvalds * fashion. 16361da177e4SLinus Torvalds * Note: 16371da177e4SLinus Torvalds * - It is assumed that the socket was locked by the caller. 16381da177e4SLinus Torvalds * - The routine does not block. 16391da177e4SLinus Torvalds * - At present, there is no support for reading OOB data 16401da177e4SLinus Torvalds * or for 'peeking' the socket using this routine 16411da177e4SLinus Torvalds * (although both would be easy to implement). 16421da177e4SLinus Torvalds */ 16431da177e4SLinus Torvalds int tcp_read_sock(struct sock *sk, read_descriptor_t *desc, 16441da177e4SLinus Torvalds sk_read_actor_t recv_actor) 16451da177e4SLinus Torvalds { 16461da177e4SLinus Torvalds struct sk_buff *skb; 16471da177e4SLinus Torvalds struct tcp_sock *tp = tcp_sk(sk); 16481da177e4SLinus Torvalds u32 seq = tp->copied_seq; 16491da177e4SLinus Torvalds u32 offset; 16501da177e4SLinus Torvalds int copied = 0; 16511da177e4SLinus Torvalds 16521da177e4SLinus Torvalds if (sk->sk_state == TCP_LISTEN) 16531da177e4SLinus Torvalds return -ENOTCONN; 16541da177e4SLinus Torvalds while ((skb = tcp_recv_skb(sk, seq, &offset)) != NULL) { 16551da177e4SLinus Torvalds if (offset < skb->len) { 1656374e7b59SOctavian Purdila int used; 1657374e7b59SOctavian Purdila size_t len; 16581da177e4SLinus Torvalds 16591da177e4SLinus Torvalds len = skb->len - offset; 16601da177e4SLinus Torvalds /* Stop reading if we hit a patch of urgent data */ 16611da177e4SLinus Torvalds if (tp->urg_data) { 16621da177e4SLinus Torvalds u32 urg_offset = tp->urg_seq - seq; 16631da177e4SLinus Torvalds if (urg_offset < len) 16641da177e4SLinus Torvalds len = urg_offset; 16651da177e4SLinus Torvalds if (!len) 16661da177e4SLinus Torvalds break; 16671da177e4SLinus Torvalds } 16681da177e4SLinus Torvalds used = recv_actor(desc, skb, offset, len); 1669ff905b1eSEric Dumazet if (used <= 0) { 1670ddb61a57SJens Axboe if (!copied) 1671ddb61a57SJens Axboe copied = used; 1672ddb61a57SJens Axboe break; 1673ddb61a57SJens Axboe } else if (used <= len) { 16741da177e4SLinus Torvalds seq += used; 16751da177e4SLinus Torvalds copied += used; 16761da177e4SLinus Torvalds offset += used; 16771da177e4SLinus Torvalds } 167802275a2eSWilly Tarreau /* If recv_actor drops the lock (e.g. TCP splice 1679293ad604SOctavian Purdila * receive) the skb pointer might be invalid when 1680293ad604SOctavian Purdila * getting here: tcp_collapse might have deleted it 1681293ad604SOctavian Purdila * while aggregating skbs from the socket queue. 1682293ad604SOctavian Purdila */ 1683293ad604SOctavian Purdila skb = tcp_recv_skb(sk, seq - 1, &offset); 168402275a2eSWilly Tarreau if (!skb) 16851da177e4SLinus Torvalds break; 168602275a2eSWilly Tarreau /* TCP coalescing might have appended data to the skb. 168702275a2eSWilly Tarreau * Try to splice more frags 168802275a2eSWilly Tarreau */ 168902275a2eSWilly Tarreau if (offset + 1 != skb->len) 169002275a2eSWilly Tarreau continue; 16911da177e4SLinus Torvalds } 1692e11ecddfSEric Dumazet if (TCP_SKB_CB(skb)->tcp_flags & TCPHDR_FIN) { 16937bced397SDan Williams sk_eat_skb(sk, skb); 16941da177e4SLinus Torvalds ++seq; 16951da177e4SLinus Torvalds break; 16961da177e4SLinus Torvalds } 16977bced397SDan Williams sk_eat_skb(sk, skb); 16981da177e4SLinus Torvalds if (!desc->count) 16991da177e4SLinus Torvalds break; 1700baff42abSSteven J. Magnani tp->copied_seq = seq; 17011da177e4SLinus Torvalds } 17021da177e4SLinus Torvalds tp->copied_seq = seq; 17031da177e4SLinus Torvalds 17041da177e4SLinus Torvalds tcp_rcv_space_adjust(sk); 17051da177e4SLinus Torvalds 17061da177e4SLinus Torvalds /* Clean up data we have read: This will do ACK frames. */ 1707f26845b4SEric Dumazet if (copied > 0) { 1708f26845b4SEric Dumazet tcp_recv_skb(sk, seq, &offset); 17090e4b4992SChris Leech tcp_cleanup_rbuf(sk, copied); 1710f26845b4SEric Dumazet } 17111da177e4SLinus Torvalds return copied; 17121da177e4SLinus Torvalds } 17134bc2f18bSEric Dumazet EXPORT_SYMBOL(tcp_read_sock); 17141da177e4SLinus Torvalds 171532035585STom Herbert int tcp_peek_len(struct socket *sock) 171632035585STom Herbert { 171732035585STom Herbert return tcp_inq(sock->sk); 171832035585STom Herbert } 171932035585STom Herbert EXPORT_SYMBOL(tcp_peek_len); 172032035585STom Herbert 172198aaa913SMike Maloney static void tcp_update_recv_tstamps(struct sk_buff *skb, 172298aaa913SMike Maloney struct scm_timestamping *tss) 172398aaa913SMike Maloney { 172498aaa913SMike Maloney if (skb->tstamp) 172598aaa913SMike Maloney tss->ts[0] = ktime_to_timespec(skb->tstamp); 172698aaa913SMike Maloney else 172798aaa913SMike Maloney tss->ts[0] = (struct timespec) {0}; 172898aaa913SMike Maloney 172998aaa913SMike Maloney if (skb_hwtstamps(skb)->hwtstamp) 173098aaa913SMike Maloney tss->ts[2] = ktime_to_timespec(skb_hwtstamps(skb)->hwtstamp); 173198aaa913SMike Maloney else 173298aaa913SMike Maloney tss->ts[2] = (struct timespec) {0}; 173398aaa913SMike Maloney } 173498aaa913SMike Maloney 173598aaa913SMike Maloney /* Similar to __sock_recv_timestamp, but does not require an skb */ 173698aaa913SMike Maloney void tcp_recv_timestamp(struct msghdr *msg, const struct sock *sk, 173798aaa913SMike Maloney struct scm_timestamping *tss) 173898aaa913SMike Maloney { 173998aaa913SMike Maloney struct timeval tv; 174098aaa913SMike Maloney bool has_timestamping = false; 174198aaa913SMike Maloney 174298aaa913SMike Maloney if (tss->ts[0].tv_sec || tss->ts[0].tv_nsec) { 174398aaa913SMike Maloney if (sock_flag(sk, SOCK_RCVTSTAMP)) { 174498aaa913SMike Maloney if (sock_flag(sk, SOCK_RCVTSTAMPNS)) { 174598aaa913SMike Maloney put_cmsg(msg, SOL_SOCKET, SCM_TIMESTAMPNS, 174698aaa913SMike Maloney sizeof(tss->ts[0]), &tss->ts[0]); 174798aaa913SMike Maloney } else { 174898aaa913SMike Maloney tv.tv_sec = tss->ts[0].tv_sec; 174998aaa913SMike Maloney tv.tv_usec = tss->ts[0].tv_nsec / 1000; 175098aaa913SMike Maloney 175198aaa913SMike Maloney put_cmsg(msg, SOL_SOCKET, SCM_TIMESTAMP, 175298aaa913SMike Maloney sizeof(tv), &tv); 175398aaa913SMike Maloney } 175498aaa913SMike Maloney } 175598aaa913SMike Maloney 175698aaa913SMike Maloney if (sk->sk_tsflags & SOF_TIMESTAMPING_SOFTWARE) 175798aaa913SMike Maloney has_timestamping = true; 175898aaa913SMike Maloney else 175998aaa913SMike Maloney tss->ts[0] = (struct timespec) {0}; 176098aaa913SMike Maloney } 176198aaa913SMike Maloney 176298aaa913SMike Maloney if (tss->ts[2].tv_sec || tss->ts[2].tv_nsec) { 176398aaa913SMike Maloney if (sk->sk_tsflags & SOF_TIMESTAMPING_RAW_HARDWARE) 176498aaa913SMike Maloney has_timestamping = true; 176598aaa913SMike Maloney else 176698aaa913SMike Maloney tss->ts[2] = (struct timespec) {0}; 176798aaa913SMike Maloney } 176898aaa913SMike Maloney 176998aaa913SMike Maloney if (has_timestamping) { 177098aaa913SMike Maloney tss->ts[1] = (struct timespec) {0}; 177198aaa913SMike Maloney put_cmsg(msg, SOL_SOCKET, SCM_TIMESTAMPING, 177298aaa913SMike Maloney sizeof(*tss), tss); 177398aaa913SMike Maloney } 177498aaa913SMike Maloney } 177598aaa913SMike Maloney 17761da177e4SLinus Torvalds /* 17771da177e4SLinus Torvalds * This routine copies from a sock struct into the user buffer. 17781da177e4SLinus Torvalds * 17791da177e4SLinus Torvalds * Technical note: in 2.3 we work on _locked_ socket, so that 17801da177e4SLinus Torvalds * tricks with *seq access order and skb->users are not required. 17811da177e4SLinus Torvalds * Probably, code can be easily improved even more. 17821da177e4SLinus Torvalds */ 17831da177e4SLinus Torvalds 17841b784140SYing Xue int tcp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len, int nonblock, 17851b784140SYing Xue int flags, int *addr_len) 17861da177e4SLinus Torvalds { 17871da177e4SLinus Torvalds struct tcp_sock *tp = tcp_sk(sk); 17881da177e4SLinus Torvalds int copied = 0; 17891da177e4SLinus Torvalds u32 peek_seq; 17901da177e4SLinus Torvalds u32 *seq; 17911da177e4SLinus Torvalds unsigned long used; 17921da177e4SLinus Torvalds int err; 17931da177e4SLinus Torvalds int target; /* Read at least this many bytes */ 17941da177e4SLinus Torvalds long timeo; 1795dfbafc99SSabrina Dubroca struct sk_buff *skb, *last; 179677527313SIlpo Järvinen u32 urg_hole = 0; 179798aaa913SMike Maloney struct scm_timestamping tss; 179898aaa913SMike Maloney bool has_tss = false; 17991da177e4SLinus Torvalds 18004ed2d765SWillem de Bruijn if (unlikely(flags & MSG_ERRQUEUE)) 1801f4713a3dSWillem de Bruijn return inet_recv_error(sk, msg, len, addr_len); 18024ed2d765SWillem de Bruijn 1803cbf55001SEliezer Tamir if (sk_can_busy_loop(sk) && skb_queue_empty(&sk->sk_receive_queue) && 1804cbf55001SEliezer Tamir (sk->sk_state == TCP_ESTABLISHED)) 1805cbf55001SEliezer Tamir sk_busy_loop(sk, nonblock); 1806d30e383bSEliezer Tamir 18071da177e4SLinus Torvalds lock_sock(sk); 18081da177e4SLinus Torvalds 18091da177e4SLinus Torvalds err = -ENOTCONN; 18101da177e4SLinus Torvalds if (sk->sk_state == TCP_LISTEN) 18111da177e4SLinus Torvalds goto out; 18121da177e4SLinus Torvalds 18131da177e4SLinus Torvalds timeo = sock_rcvtimeo(sk, nonblock); 18141da177e4SLinus Torvalds 18151da177e4SLinus Torvalds /* Urgent data needs to be handled specially. */ 18161da177e4SLinus Torvalds if (flags & MSG_OOB) 18171da177e4SLinus Torvalds goto recv_urg; 18181da177e4SLinus Torvalds 1819c0e88ff0SPavel Emelyanov if (unlikely(tp->repair)) { 1820c0e88ff0SPavel Emelyanov err = -EPERM; 1821c0e88ff0SPavel Emelyanov if (!(flags & MSG_PEEK)) 1822c0e88ff0SPavel Emelyanov goto out; 1823c0e88ff0SPavel Emelyanov 1824c0e88ff0SPavel Emelyanov if (tp->repair_queue == TCP_SEND_QUEUE) 1825c0e88ff0SPavel Emelyanov goto recv_sndq; 1826c0e88ff0SPavel Emelyanov 1827c0e88ff0SPavel Emelyanov err = -EINVAL; 1828c0e88ff0SPavel Emelyanov if (tp->repair_queue == TCP_NO_QUEUE) 1829c0e88ff0SPavel Emelyanov goto out; 1830c0e88ff0SPavel Emelyanov 1831c0e88ff0SPavel Emelyanov /* 'common' recv queue MSG_PEEK-ing */ 1832c0e88ff0SPavel Emelyanov } 1833c0e88ff0SPavel Emelyanov 18341da177e4SLinus Torvalds seq = &tp->copied_seq; 18351da177e4SLinus Torvalds if (flags & MSG_PEEK) { 18361da177e4SLinus Torvalds peek_seq = tp->copied_seq; 18371da177e4SLinus Torvalds seq = &peek_seq; 18381da177e4SLinus Torvalds } 18391da177e4SLinus Torvalds 18401da177e4SLinus Torvalds target = sock_rcvlowat(sk, flags & MSG_WAITALL, len); 18411da177e4SLinus Torvalds 18421da177e4SLinus Torvalds do { 18431da177e4SLinus Torvalds u32 offset; 18441da177e4SLinus Torvalds 18451da177e4SLinus Torvalds /* Are we at urgent data? Stop if we have read anything or have SIGURG pending. */ 18461da177e4SLinus Torvalds if (tp->urg_data && tp->urg_seq == *seq) { 18471da177e4SLinus Torvalds if (copied) 18481da177e4SLinus Torvalds break; 18491da177e4SLinus Torvalds if (signal_pending(current)) { 18501da177e4SLinus Torvalds copied = timeo ? sock_intr_errno(timeo) : -EAGAIN; 18511da177e4SLinus Torvalds break; 18521da177e4SLinus Torvalds } 18531da177e4SLinus Torvalds } 18541da177e4SLinus Torvalds 18551da177e4SLinus Torvalds /* Next get a buffer. */ 18561da177e4SLinus Torvalds 1857dfbafc99SSabrina Dubroca last = skb_peek_tail(&sk->sk_receive_queue); 185891521944SDavid S. Miller skb_queue_walk(&sk->sk_receive_queue, skb) { 1859dfbafc99SSabrina Dubroca last = skb; 18601da177e4SLinus Torvalds /* Now that we have two receive queues this 18611da177e4SLinus Torvalds * shouldn't happen. 18621da177e4SLinus Torvalds */ 1863d792c100SIlpo Järvinen if (WARN(before(*seq, TCP_SKB_CB(skb)->seq), 18642af6fd8bSJoe Perches "recvmsg bug: copied %X seq %X rcvnxt %X fl %X\n", 18652af6fd8bSJoe Perches *seq, TCP_SKB_CB(skb)->seq, tp->rcv_nxt, 1866d792c100SIlpo Järvinen flags)) 18671da177e4SLinus Torvalds break; 1868d792c100SIlpo Järvinen 18691da177e4SLinus Torvalds offset = *seq - TCP_SKB_CB(skb)->seq; 18709d691539SEric Dumazet if (unlikely(TCP_SKB_CB(skb)->tcp_flags & TCPHDR_SYN)) { 18719d691539SEric Dumazet pr_err_once("%s: found a SYN, please report !\n", __func__); 18721da177e4SLinus Torvalds offset--; 18739d691539SEric Dumazet } 18741da177e4SLinus Torvalds if (offset < skb->len) 18751da177e4SLinus Torvalds goto found_ok_skb; 1876e11ecddfSEric Dumazet if (TCP_SKB_CB(skb)->tcp_flags & TCPHDR_FIN) 18771da177e4SLinus Torvalds goto found_fin_ok; 18782af6fd8bSJoe Perches WARN(!(flags & MSG_PEEK), 18792af6fd8bSJoe Perches "recvmsg bug 2: copied %X seq %X rcvnxt %X fl %X\n", 18802af6fd8bSJoe Perches *seq, TCP_SKB_CB(skb)->seq, tp->rcv_nxt, flags); 188191521944SDavid S. Miller } 18821da177e4SLinus Torvalds 18831da177e4SLinus Torvalds /* Well, if we have backlog, try to process it now yet. */ 18841da177e4SLinus Torvalds 18851da177e4SLinus Torvalds if (copied >= target && !sk->sk_backlog.tail) 18861da177e4SLinus Torvalds break; 18871da177e4SLinus Torvalds 18881da177e4SLinus Torvalds if (copied) { 18891da177e4SLinus Torvalds if (sk->sk_err || 18901da177e4SLinus Torvalds sk->sk_state == TCP_CLOSE || 18911da177e4SLinus Torvalds (sk->sk_shutdown & RCV_SHUTDOWN) || 18921da177e4SLinus Torvalds !timeo || 1893518a09efSDavid S. Miller signal_pending(current)) 18941da177e4SLinus Torvalds break; 18951da177e4SLinus Torvalds } else { 18961da177e4SLinus Torvalds if (sock_flag(sk, SOCK_DONE)) 18971da177e4SLinus Torvalds break; 18981da177e4SLinus Torvalds 18991da177e4SLinus Torvalds if (sk->sk_err) { 19001da177e4SLinus Torvalds copied = sock_error(sk); 19011da177e4SLinus Torvalds break; 19021da177e4SLinus Torvalds } 19031da177e4SLinus Torvalds 19041da177e4SLinus Torvalds if (sk->sk_shutdown & RCV_SHUTDOWN) 19051da177e4SLinus Torvalds break; 19061da177e4SLinus Torvalds 19071da177e4SLinus Torvalds if (sk->sk_state == TCP_CLOSE) { 19081da177e4SLinus Torvalds if (!sock_flag(sk, SOCK_DONE)) { 19091da177e4SLinus Torvalds /* This occurs when user tries to read 19101da177e4SLinus Torvalds * from never connected socket. 19111da177e4SLinus Torvalds */ 19121da177e4SLinus Torvalds copied = -ENOTCONN; 19131da177e4SLinus Torvalds break; 19141da177e4SLinus Torvalds } 19151da177e4SLinus Torvalds break; 19161da177e4SLinus Torvalds } 19171da177e4SLinus Torvalds 19181da177e4SLinus Torvalds if (!timeo) { 19191da177e4SLinus Torvalds copied = -EAGAIN; 19201da177e4SLinus Torvalds break; 19211da177e4SLinus Torvalds } 19221da177e4SLinus Torvalds 19231da177e4SLinus Torvalds if (signal_pending(current)) { 19241da177e4SLinus Torvalds copied = sock_intr_errno(timeo); 19251da177e4SLinus Torvalds break; 19261da177e4SLinus Torvalds } 19271da177e4SLinus Torvalds } 19281da177e4SLinus Torvalds 19290e4b4992SChris Leech tcp_cleanup_rbuf(sk, copied); 19301da177e4SLinus Torvalds 19311da177e4SLinus Torvalds if (copied >= target) { 19321da177e4SLinus Torvalds /* Do not sleep, just process backlog. */ 19331da177e4SLinus Torvalds release_sock(sk); 19341da177e4SLinus Torvalds lock_sock(sk); 1935dfbafc99SSabrina Dubroca } else { 1936dfbafc99SSabrina Dubroca sk_wait_data(sk, &timeo, last); 1937dfbafc99SSabrina Dubroca } 19381da177e4SLinus Torvalds 193977527313SIlpo Järvinen if ((flags & MSG_PEEK) && 194077527313SIlpo Järvinen (peek_seq - copied - urg_hole != tp->copied_seq)) { 1941e87cc472SJoe Perches net_dbg_ratelimited("TCP(%s:%d): Application bug, race in MSG_PEEK\n", 1942e87cc472SJoe Perches current->comm, 1943e87cc472SJoe Perches task_pid_nr(current)); 19441da177e4SLinus Torvalds peek_seq = tp->copied_seq; 19451da177e4SLinus Torvalds } 19461da177e4SLinus Torvalds continue; 19471da177e4SLinus Torvalds 19481da177e4SLinus Torvalds found_ok_skb: 19491da177e4SLinus Torvalds /* Ok so how much can we use? */ 19501da177e4SLinus Torvalds used = skb->len - offset; 19511da177e4SLinus Torvalds if (len < used) 19521da177e4SLinus Torvalds used = len; 19531da177e4SLinus Torvalds 19541da177e4SLinus Torvalds /* Do we have urgent data here? */ 19551da177e4SLinus Torvalds if (tp->urg_data) { 19561da177e4SLinus Torvalds u32 urg_offset = tp->urg_seq - *seq; 19571da177e4SLinus Torvalds if (urg_offset < used) { 19581da177e4SLinus Torvalds if (!urg_offset) { 19591da177e4SLinus Torvalds if (!sock_flag(sk, SOCK_URGINLINE)) { 19601da177e4SLinus Torvalds ++*seq; 196177527313SIlpo Järvinen urg_hole++; 19621da177e4SLinus Torvalds offset++; 19631da177e4SLinus Torvalds used--; 19641da177e4SLinus Torvalds if (!used) 19651da177e4SLinus Torvalds goto skip_copy; 19661da177e4SLinus Torvalds } 19671da177e4SLinus Torvalds } else 19681da177e4SLinus Torvalds used = urg_offset; 19691da177e4SLinus Torvalds } 19701da177e4SLinus Torvalds } 19711da177e4SLinus Torvalds 19721da177e4SLinus Torvalds if (!(flags & MSG_TRUNC)) { 197351f3d02bSDavid S. Miller err = skb_copy_datagram_msg(skb, offset, msg, used); 19741da177e4SLinus Torvalds if (err) { 19751da177e4SLinus Torvalds /* Exception. Bailout! */ 19761da177e4SLinus Torvalds if (!copied) 19771da177e4SLinus Torvalds copied = -EFAULT; 19781da177e4SLinus Torvalds break; 19791da177e4SLinus Torvalds } 19801da177e4SLinus Torvalds } 19811da177e4SLinus Torvalds 19821da177e4SLinus Torvalds *seq += used; 19831da177e4SLinus Torvalds copied += used; 19841da177e4SLinus Torvalds len -= used; 19851da177e4SLinus Torvalds 19861da177e4SLinus Torvalds tcp_rcv_space_adjust(sk); 19871da177e4SLinus Torvalds 19881da177e4SLinus Torvalds skip_copy: 198931770e34SFlorian Westphal if (tp->urg_data && after(tp->copied_seq, tp->urg_seq)) { 19901da177e4SLinus Torvalds tp->urg_data = 0; 199131770e34SFlorian Westphal tcp_fast_path_check(sk); 199231770e34SFlorian Westphal } 19931da177e4SLinus Torvalds if (used + offset < skb->len) 19941da177e4SLinus Torvalds continue; 19951da177e4SLinus Torvalds 199698aaa913SMike Maloney if (TCP_SKB_CB(skb)->has_rxtstamp) { 199798aaa913SMike Maloney tcp_update_recv_tstamps(skb, &tss); 199898aaa913SMike Maloney has_tss = true; 199998aaa913SMike Maloney } 2000e11ecddfSEric Dumazet if (TCP_SKB_CB(skb)->tcp_flags & TCPHDR_FIN) 20011da177e4SLinus Torvalds goto found_fin_ok; 20027bced397SDan Williams if (!(flags & MSG_PEEK)) 20037bced397SDan Williams sk_eat_skb(sk, skb); 20041da177e4SLinus Torvalds continue; 20051da177e4SLinus Torvalds 20061da177e4SLinus Torvalds found_fin_ok: 20071da177e4SLinus Torvalds /* Process the FIN. */ 20081da177e4SLinus Torvalds ++*seq; 20097bced397SDan Williams if (!(flags & MSG_PEEK)) 20107bced397SDan Williams sk_eat_skb(sk, skb); 20111da177e4SLinus Torvalds break; 20121da177e4SLinus Torvalds } while (len > 0); 20131da177e4SLinus Torvalds 20141da177e4SLinus Torvalds /* According to UNIX98, msg_name/msg_namelen are ignored 20151da177e4SLinus Torvalds * on connected socket. I was just happy when found this 8) --ANK 20161da177e4SLinus Torvalds */ 20171da177e4SLinus Torvalds 201898aaa913SMike Maloney if (has_tss) 201998aaa913SMike Maloney tcp_recv_timestamp(msg, sk, &tss); 202098aaa913SMike Maloney 20211da177e4SLinus Torvalds /* Clean up data we have read: This will do ACK frames. */ 20220e4b4992SChris Leech tcp_cleanup_rbuf(sk, copied); 20231da177e4SLinus Torvalds 20241da177e4SLinus Torvalds release_sock(sk); 20251da177e4SLinus Torvalds return copied; 20261da177e4SLinus Torvalds 20271da177e4SLinus Torvalds out: 20281da177e4SLinus Torvalds release_sock(sk); 20291da177e4SLinus Torvalds return err; 20301da177e4SLinus Torvalds 20311da177e4SLinus Torvalds recv_urg: 2032377f0a08SRami Rosen err = tcp_recv_urg(sk, msg, len, flags); 20331da177e4SLinus Torvalds goto out; 2034c0e88ff0SPavel Emelyanov 2035c0e88ff0SPavel Emelyanov recv_sndq: 2036c0e88ff0SPavel Emelyanov err = tcp_peek_sndq(sk, msg, len); 2037c0e88ff0SPavel Emelyanov goto out; 20381da177e4SLinus Torvalds } 20394bc2f18bSEric Dumazet EXPORT_SYMBOL(tcp_recvmsg); 20401da177e4SLinus Torvalds 2041490d5046SIlpo Järvinen void tcp_set_state(struct sock *sk, int state) 2042490d5046SIlpo Järvinen { 2043490d5046SIlpo Järvinen int oldstate = sk->sk_state; 2044490d5046SIlpo Järvinen 2045e8fce239SSong Liu trace_tcp_set_state(sk, oldstate, state); 2046e8fce239SSong Liu 2047490d5046SIlpo Järvinen switch (state) { 2048490d5046SIlpo Järvinen case TCP_ESTABLISHED: 2049490d5046SIlpo Järvinen if (oldstate != TCP_ESTABLISHED) 205081cc8a75SPavel Emelyanov TCP_INC_STATS(sock_net(sk), TCP_MIB_CURRESTAB); 2051490d5046SIlpo Järvinen break; 2052490d5046SIlpo Järvinen 2053490d5046SIlpo Järvinen case TCP_CLOSE: 2054490d5046SIlpo Järvinen if (oldstate == TCP_CLOSE_WAIT || oldstate == TCP_ESTABLISHED) 205581cc8a75SPavel Emelyanov TCP_INC_STATS(sock_net(sk), TCP_MIB_ESTABRESETS); 2056490d5046SIlpo Järvinen 2057490d5046SIlpo Järvinen sk->sk_prot->unhash(sk); 2058490d5046SIlpo Järvinen if (inet_csk(sk)->icsk_bind_hash && 2059490d5046SIlpo Järvinen !(sk->sk_userlocks & SOCK_BINDPORT_LOCK)) 2060ab1e0a13SArnaldo Carvalho de Melo inet_put_port(sk); 2061490d5046SIlpo Järvinen /* fall through */ 2062490d5046SIlpo Järvinen default: 2063490d5046SIlpo Järvinen if (oldstate == TCP_ESTABLISHED) 206474688e48SPavel Emelyanov TCP_DEC_STATS(sock_net(sk), TCP_MIB_CURRESTAB); 2065490d5046SIlpo Järvinen } 2066490d5046SIlpo Järvinen 2067490d5046SIlpo Järvinen /* Change state AFTER socket is unhashed to avoid closed 2068490d5046SIlpo Järvinen * socket sitting in hash tables. 2069490d5046SIlpo Järvinen */ 207000fd38d9SEric Dumazet sk_state_store(sk, state); 2071490d5046SIlpo Järvinen 2072490d5046SIlpo Järvinen #ifdef STATE_TRACE 2073490d5046SIlpo Järvinen SOCK_DEBUG(sk, "TCP sk=%p, State %s -> %s\n", sk, statename[oldstate], statename[state]); 2074490d5046SIlpo Järvinen #endif 2075490d5046SIlpo Järvinen } 2076490d5046SIlpo Järvinen EXPORT_SYMBOL_GPL(tcp_set_state); 2077490d5046SIlpo Järvinen 20781da177e4SLinus Torvalds /* 20791da177e4SLinus Torvalds * State processing on a close. This implements the state shift for 20801da177e4SLinus Torvalds * sending our FIN frame. Note that we only send a FIN for some 20811da177e4SLinus Torvalds * states. A shutdown() may have already sent the FIN, or we may be 20821da177e4SLinus Torvalds * closed. 20831da177e4SLinus Torvalds */ 20841da177e4SLinus Torvalds 20859b5b5cffSArjan van de Ven static const unsigned char new_state[16] = { 20861da177e4SLinus Torvalds /* current state: new state: action: */ 20870980c1e3SEric Dumazet [0 /* (Invalid) */] = TCP_CLOSE, 20880980c1e3SEric Dumazet [TCP_ESTABLISHED] = TCP_FIN_WAIT1 | TCP_ACTION_FIN, 20890980c1e3SEric Dumazet [TCP_SYN_SENT] = TCP_CLOSE, 20900980c1e3SEric Dumazet [TCP_SYN_RECV] = TCP_FIN_WAIT1 | TCP_ACTION_FIN, 20910980c1e3SEric Dumazet [TCP_FIN_WAIT1] = TCP_FIN_WAIT1, 20920980c1e3SEric Dumazet [TCP_FIN_WAIT2] = TCP_FIN_WAIT2, 20930980c1e3SEric Dumazet [TCP_TIME_WAIT] = TCP_CLOSE, 20940980c1e3SEric Dumazet [TCP_CLOSE] = TCP_CLOSE, 20950980c1e3SEric Dumazet [TCP_CLOSE_WAIT] = TCP_LAST_ACK | TCP_ACTION_FIN, 20960980c1e3SEric Dumazet [TCP_LAST_ACK] = TCP_LAST_ACK, 20970980c1e3SEric Dumazet [TCP_LISTEN] = TCP_CLOSE, 20980980c1e3SEric Dumazet [TCP_CLOSING] = TCP_CLOSING, 20990980c1e3SEric Dumazet [TCP_NEW_SYN_RECV] = TCP_CLOSE, /* should not happen ! */ 21001da177e4SLinus Torvalds }; 21011da177e4SLinus Torvalds 21021da177e4SLinus Torvalds static int tcp_close_state(struct sock *sk) 21031da177e4SLinus Torvalds { 21041da177e4SLinus Torvalds int next = (int)new_state[sk->sk_state]; 21051da177e4SLinus Torvalds int ns = next & TCP_STATE_MASK; 21061da177e4SLinus Torvalds 21071da177e4SLinus Torvalds tcp_set_state(sk, ns); 21081da177e4SLinus Torvalds 21091da177e4SLinus Torvalds return next & TCP_ACTION_FIN; 21101da177e4SLinus Torvalds } 21111da177e4SLinus Torvalds 21121da177e4SLinus Torvalds /* 21131da177e4SLinus Torvalds * Shutdown the sending side of a connection. Much like close except 21141f29b058SSatoru SATOH * that we don't receive shut down or sock_set_flag(sk, SOCK_DEAD). 21151da177e4SLinus Torvalds */ 21161da177e4SLinus Torvalds 21171da177e4SLinus Torvalds void tcp_shutdown(struct sock *sk, int how) 21181da177e4SLinus Torvalds { 21191da177e4SLinus Torvalds /* We need to grab some memory, and put together a FIN, 21201da177e4SLinus Torvalds * and then put it into the queue to be sent. 21211da177e4SLinus Torvalds * Tim MacKenzie(tym@dibbler.cs.monash.edu.au) 4 Dec '92. 21221da177e4SLinus Torvalds */ 21231da177e4SLinus Torvalds if (!(how & SEND_SHUTDOWN)) 21241da177e4SLinus Torvalds return; 21251da177e4SLinus Torvalds 21261da177e4SLinus Torvalds /* If we've already sent a FIN, or it's a closed state, skip this. */ 21271da177e4SLinus Torvalds if ((1 << sk->sk_state) & 21281da177e4SLinus Torvalds (TCPF_ESTABLISHED | TCPF_SYN_SENT | 21291da177e4SLinus Torvalds TCPF_SYN_RECV | TCPF_CLOSE_WAIT)) { 21301da177e4SLinus Torvalds /* Clear out any half completed packets. FIN if needed. */ 21311da177e4SLinus Torvalds if (tcp_close_state(sk)) 21321da177e4SLinus Torvalds tcp_send_fin(sk); 21331da177e4SLinus Torvalds } 21341da177e4SLinus Torvalds } 21354bc2f18bSEric Dumazet EXPORT_SYMBOL(tcp_shutdown); 21361da177e4SLinus Torvalds 2137efcdbf24SArun Sharma bool tcp_check_oom(struct sock *sk, int shift) 2138efcdbf24SArun Sharma { 2139efcdbf24SArun Sharma bool too_many_orphans, out_of_socket_memory; 2140efcdbf24SArun Sharma 2141efcdbf24SArun Sharma too_many_orphans = tcp_too_many_orphans(sk, shift); 2142efcdbf24SArun Sharma out_of_socket_memory = tcp_out_of_memory(sk); 2143efcdbf24SArun Sharma 2144e87cc472SJoe Perches if (too_many_orphans) 2145e87cc472SJoe Perches net_info_ratelimited("too many orphaned sockets\n"); 2146e87cc472SJoe Perches if (out_of_socket_memory) 2147e87cc472SJoe Perches net_info_ratelimited("out of memory -- consider tuning tcp_mem\n"); 2148efcdbf24SArun Sharma return too_many_orphans || out_of_socket_memory; 2149efcdbf24SArun Sharma } 2150efcdbf24SArun Sharma 21511da177e4SLinus Torvalds void tcp_close(struct sock *sk, long timeout) 21521da177e4SLinus Torvalds { 21531da177e4SLinus Torvalds struct sk_buff *skb; 21541da177e4SLinus Torvalds int data_was_unread = 0; 215575c2d907SHerbert Xu int state; 21561da177e4SLinus Torvalds 21571da177e4SLinus Torvalds lock_sock(sk); 21581da177e4SLinus Torvalds sk->sk_shutdown = SHUTDOWN_MASK; 21591da177e4SLinus Torvalds 21601da177e4SLinus Torvalds if (sk->sk_state == TCP_LISTEN) { 21611da177e4SLinus Torvalds tcp_set_state(sk, TCP_CLOSE); 21621da177e4SLinus Torvalds 21631da177e4SLinus Torvalds /* Special case. */ 21640a5578cfSArnaldo Carvalho de Melo inet_csk_listen_stop(sk); 21651da177e4SLinus Torvalds 21661da177e4SLinus Torvalds goto adjudge_to_death; 21671da177e4SLinus Torvalds } 21681da177e4SLinus Torvalds 21691da177e4SLinus Torvalds /* We need to flush the recv. buffs. We do this only on the 21701da177e4SLinus Torvalds * descriptor close, not protocol-sourced closes, because the 21711da177e4SLinus Torvalds * reader process may not have drained the data yet! 21721da177e4SLinus Torvalds */ 21731da177e4SLinus Torvalds while ((skb = __skb_dequeue(&sk->sk_receive_queue)) != NULL) { 2174e11ecddfSEric Dumazet u32 len = TCP_SKB_CB(skb)->end_seq - TCP_SKB_CB(skb)->seq; 2175e11ecddfSEric Dumazet 2176e11ecddfSEric Dumazet if (TCP_SKB_CB(skb)->tcp_flags & TCPHDR_FIN) 2177e11ecddfSEric Dumazet len--; 21781da177e4SLinus Torvalds data_was_unread += len; 21791da177e4SLinus Torvalds __kfree_skb(skb); 21801da177e4SLinus Torvalds } 21811da177e4SLinus Torvalds 21823ab224beSHideo Aoki sk_mem_reclaim(sk); 21831da177e4SLinus Torvalds 2184565b7b2dSKonstantin Khorenko /* If socket has been already reset (e.g. in tcp_reset()) - kill it. */ 2185565b7b2dSKonstantin Khorenko if (sk->sk_state == TCP_CLOSE) 2186565b7b2dSKonstantin Khorenko goto adjudge_to_death; 2187565b7b2dSKonstantin Khorenko 218865bb723cSGerrit Renker /* As outlined in RFC 2525, section 2.17, we send a RST here because 218965bb723cSGerrit Renker * data was lost. To witness the awful effects of the old behavior of 219065bb723cSGerrit Renker * always doing a FIN, run an older 2.1.x kernel or 2.0.x, start a bulk 219165bb723cSGerrit Renker * GET in an FTP client, suspend the process, wait for the client to 219265bb723cSGerrit Renker * advertise a zero window, then kill -9 the FTP client, wheee... 219365bb723cSGerrit Renker * Note: timeout is always zero in such a case. 21941da177e4SLinus Torvalds */ 2195ee995283SPavel Emelyanov if (unlikely(tcp_sk(sk)->repair)) { 2196ee995283SPavel Emelyanov sk->sk_prot->disconnect(sk, 0); 2197ee995283SPavel Emelyanov } else if (data_was_unread) { 21981da177e4SLinus Torvalds /* Unread data was tossed, zap the connection. */ 21996aef70a8SEric Dumazet NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPABORTONCLOSE); 22001da177e4SLinus Torvalds tcp_set_state(sk, TCP_CLOSE); 2201aa133076SWu Fengguang tcp_send_active_reset(sk, sk->sk_allocation); 22021da177e4SLinus Torvalds } else if (sock_flag(sk, SOCK_LINGER) && !sk->sk_lingertime) { 22031da177e4SLinus Torvalds /* Check zero linger _after_ checking for unread data. */ 22041da177e4SLinus Torvalds sk->sk_prot->disconnect(sk, 0); 22056aef70a8SEric Dumazet NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPABORTONDATA); 22061da177e4SLinus Torvalds } else if (tcp_close_state(sk)) { 22071da177e4SLinus Torvalds /* We FIN if the application ate all the data before 22081da177e4SLinus Torvalds * zapping the connection. 22091da177e4SLinus Torvalds */ 22101da177e4SLinus Torvalds 22111da177e4SLinus Torvalds /* RED-PEN. Formally speaking, we have broken TCP state 22121da177e4SLinus Torvalds * machine. State transitions: 22131da177e4SLinus Torvalds * 22141da177e4SLinus Torvalds * TCP_ESTABLISHED -> TCP_FIN_WAIT1 22151da177e4SLinus Torvalds * TCP_SYN_RECV -> TCP_FIN_WAIT1 (forget it, it's impossible) 22161da177e4SLinus Torvalds * TCP_CLOSE_WAIT -> TCP_LAST_ACK 22171da177e4SLinus Torvalds * 22181da177e4SLinus Torvalds * are legal only when FIN has been sent (i.e. in window), 22191da177e4SLinus Torvalds * rather than queued out of window. Purists blame. 22201da177e4SLinus Torvalds * 22211da177e4SLinus Torvalds * F.e. "RFC state" is ESTABLISHED, 22221da177e4SLinus Torvalds * if Linux state is FIN-WAIT-1, but FIN is still not sent. 22231da177e4SLinus Torvalds * 22241da177e4SLinus Torvalds * The visible declinations are that sometimes 22251da177e4SLinus Torvalds * we enter time-wait state, when it is not required really 22261da177e4SLinus Torvalds * (harmless), do not send active resets, when they are 22271da177e4SLinus Torvalds * required by specs (TCP_ESTABLISHED, TCP_CLOSE_WAIT, when 22281da177e4SLinus Torvalds * they look as CLOSING or LAST_ACK for Linux) 22291da177e4SLinus Torvalds * Probably, I missed some more holelets. 22301da177e4SLinus Torvalds * --ANK 22318336886fSJerry Chu * XXX (TFO) - To start off we don't support SYN+ACK+FIN 22328336886fSJerry Chu * in a single packet! (May consider it later but will 22338336886fSJerry Chu * probably need API support or TCP_CORK SYN-ACK until 22348336886fSJerry Chu * data is written and socket is closed.) 22351da177e4SLinus Torvalds */ 22361da177e4SLinus Torvalds tcp_send_fin(sk); 22371da177e4SLinus Torvalds } 22381da177e4SLinus Torvalds 22391da177e4SLinus Torvalds sk_stream_wait_close(sk, timeout); 22401da177e4SLinus Torvalds 22411da177e4SLinus Torvalds adjudge_to_death: 224275c2d907SHerbert Xu state = sk->sk_state; 224375c2d907SHerbert Xu sock_hold(sk); 224475c2d907SHerbert Xu sock_orphan(sk); 224575c2d907SHerbert Xu 22461da177e4SLinus Torvalds /* It is the last release_sock in its life. It will remove backlog. */ 22471da177e4SLinus Torvalds release_sock(sk); 22481da177e4SLinus Torvalds 22491da177e4SLinus Torvalds 22501da177e4SLinus Torvalds /* Now socket is owned by kernel and we acquire BH lock 2251a777f715SRohit Chavan * to finish close. No need to check for user refs. 22521da177e4SLinus Torvalds */ 22531da177e4SLinus Torvalds local_bh_disable(); 22541da177e4SLinus Torvalds bh_lock_sock(sk); 2255547b792cSIlpo Järvinen WARN_ON(sock_owned_by_user(sk)); 22561da177e4SLinus Torvalds 2257eb4dea58SHerbert Xu percpu_counter_inc(sk->sk_prot->orphan_count); 2258eb4dea58SHerbert Xu 225975c2d907SHerbert Xu /* Have we already been destroyed by a softirq or backlog? */ 226075c2d907SHerbert Xu if (state != TCP_CLOSE && sk->sk_state == TCP_CLOSE) 226175c2d907SHerbert Xu goto out; 22621da177e4SLinus Torvalds 22631da177e4SLinus Torvalds /* This is a (useful) BSD violating of the RFC. There is a 22641da177e4SLinus Torvalds * problem with TCP as specified in that the other end could 22651da177e4SLinus Torvalds * keep a socket open forever with no application left this end. 2266b10bd54cSJesper Juhl * We use a 1 minute timeout (about the same as BSD) then kill 22671da177e4SLinus Torvalds * our end. If they send after that then tough - BUT: long enough 22681da177e4SLinus Torvalds * that we won't make the old 4*rto = almost no time - whoops 22691da177e4SLinus Torvalds * reset mistake. 22701da177e4SLinus Torvalds * 22711da177e4SLinus Torvalds * Nope, it was not mistake. It is really desired behaviour 22721da177e4SLinus Torvalds * f.e. on http servers, when such sockets are useless, but 22731da177e4SLinus Torvalds * consume significant resources. Let's do it with special 22741da177e4SLinus Torvalds * linger2 option. --ANK 22751da177e4SLinus Torvalds */ 22761da177e4SLinus Torvalds 22771da177e4SLinus Torvalds if (sk->sk_state == TCP_FIN_WAIT2) { 22781da177e4SLinus Torvalds struct tcp_sock *tp = tcp_sk(sk); 22791da177e4SLinus Torvalds if (tp->linger2 < 0) { 22801da177e4SLinus Torvalds tcp_set_state(sk, TCP_CLOSE); 22811da177e4SLinus Torvalds tcp_send_active_reset(sk, GFP_ATOMIC); 228202a1d6e7SEric Dumazet __NET_INC_STATS(sock_net(sk), 2283de0744afSPavel Emelyanov LINUX_MIB_TCPABORTONLINGER); 22841da177e4SLinus Torvalds } else { 2285463c84b9SArnaldo Carvalho de Melo const int tmo = tcp_fin_time(sk); 22861da177e4SLinus Torvalds 22871da177e4SLinus Torvalds if (tmo > TCP_TIMEWAIT_LEN) { 228852499afeSDavid S. Miller inet_csk_reset_keepalive_timer(sk, 228952499afeSDavid S. Miller tmo - TCP_TIMEWAIT_LEN); 22901da177e4SLinus Torvalds } else { 22911da177e4SLinus Torvalds tcp_time_wait(sk, TCP_FIN_WAIT2, tmo); 22921da177e4SLinus Torvalds goto out; 22931da177e4SLinus Torvalds } 22941da177e4SLinus Torvalds } 22951da177e4SLinus Torvalds } 22961da177e4SLinus Torvalds if (sk->sk_state != TCP_CLOSE) { 22973ab224beSHideo Aoki sk_mem_reclaim(sk); 2298efcdbf24SArun Sharma if (tcp_check_oom(sk, 0)) { 22991da177e4SLinus Torvalds tcp_set_state(sk, TCP_CLOSE); 23001da177e4SLinus Torvalds tcp_send_active_reset(sk, GFP_ATOMIC); 230102a1d6e7SEric Dumazet __NET_INC_STATS(sock_net(sk), 2302de0744afSPavel Emelyanov LINUX_MIB_TCPABORTONMEMORY); 23031da177e4SLinus Torvalds } 23041da177e4SLinus Torvalds } 23051da177e4SLinus Torvalds 23068336886fSJerry Chu if (sk->sk_state == TCP_CLOSE) { 23078336886fSJerry Chu struct request_sock *req = tcp_sk(sk)->fastopen_rsk; 23088336886fSJerry Chu /* We could get here with a non-NULL req if the socket is 23098336886fSJerry Chu * aborted (e.g., closed with unread data) before 3WHS 23108336886fSJerry Chu * finishes. 23118336886fSJerry Chu */ 231200db4124SIan Morris if (req) 23138336886fSJerry Chu reqsk_fastopen_remove(sk, req, false); 23140a5578cfSArnaldo Carvalho de Melo inet_csk_destroy_sock(sk); 23158336886fSJerry Chu } 23161da177e4SLinus Torvalds /* Otherwise, socket is reprieved until protocol close. */ 23171da177e4SLinus Torvalds 23181da177e4SLinus Torvalds out: 23191da177e4SLinus Torvalds bh_unlock_sock(sk); 23201da177e4SLinus Torvalds local_bh_enable(); 23211da177e4SLinus Torvalds sock_put(sk); 23221da177e4SLinus Torvalds } 23234bc2f18bSEric Dumazet EXPORT_SYMBOL(tcp_close); 23241da177e4SLinus Torvalds 23251da177e4SLinus Torvalds /* These states need RST on ABORT according to RFC793 */ 23261da177e4SLinus Torvalds 2327a2a385d6SEric Dumazet static inline bool tcp_need_reset(int state) 23281da177e4SLinus Torvalds { 23291da177e4SLinus Torvalds return (1 << state) & 23301da177e4SLinus Torvalds (TCPF_ESTABLISHED | TCPF_CLOSE_WAIT | TCPF_FIN_WAIT1 | 23311da177e4SLinus Torvalds TCPF_FIN_WAIT2 | TCPF_SYN_RECV); 23321da177e4SLinus Torvalds } 23331da177e4SLinus Torvalds 233475c119afSEric Dumazet static void tcp_rtx_queue_purge(struct sock *sk) 233575c119afSEric Dumazet { 233675c119afSEric Dumazet struct rb_node *p = rb_first(&sk->tcp_rtx_queue); 233775c119afSEric Dumazet 233875c119afSEric Dumazet while (p) { 233975c119afSEric Dumazet struct sk_buff *skb = rb_to_skb(p); 234075c119afSEric Dumazet 234175c119afSEric Dumazet p = rb_next(p); 234275c119afSEric Dumazet /* Since we are deleting whole queue, no need to 234375c119afSEric Dumazet * list_del(&skb->tcp_tsorted_anchor) 234475c119afSEric Dumazet */ 234575c119afSEric Dumazet tcp_rtx_queue_unlink(skb, sk); 234675c119afSEric Dumazet sk_wmem_free_skb(sk, skb); 234775c119afSEric Dumazet } 234875c119afSEric Dumazet } 234975c119afSEric Dumazet 2350ac3f09baSEric Dumazet void tcp_write_queue_purge(struct sock *sk) 2351ac3f09baSEric Dumazet { 2352ac3f09baSEric Dumazet struct sk_buff *skb; 2353ac3f09baSEric Dumazet 2354ac3f09baSEric Dumazet tcp_chrono_stop(sk, TCP_CHRONO_BUSY); 2355ac3f09baSEric Dumazet while ((skb = __skb_dequeue(&sk->sk_write_queue)) != NULL) { 2356ac3f09baSEric Dumazet tcp_skb_tsorted_anchor_cleanup(skb); 2357ac3f09baSEric Dumazet sk_wmem_free_skb(sk, skb); 2358ac3f09baSEric Dumazet } 235975c119afSEric Dumazet tcp_rtx_queue_purge(sk); 2360ac3f09baSEric Dumazet INIT_LIST_HEAD(&tcp_sk(sk)->tsorted_sent_queue); 2361ac3f09baSEric Dumazet sk_mem_reclaim(sk); 2362ac3f09baSEric Dumazet tcp_clear_all_retrans_hints(tcp_sk(sk)); 2363ac3f09baSEric Dumazet } 2364ac3f09baSEric Dumazet 23651da177e4SLinus Torvalds int tcp_disconnect(struct sock *sk, int flags) 23661da177e4SLinus Torvalds { 23671da177e4SLinus Torvalds struct inet_sock *inet = inet_sk(sk); 2368463c84b9SArnaldo Carvalho de Melo struct inet_connection_sock *icsk = inet_csk(sk); 23691da177e4SLinus Torvalds struct tcp_sock *tp = tcp_sk(sk); 23701da177e4SLinus Torvalds int err = 0; 23711da177e4SLinus Torvalds int old_state = sk->sk_state; 23721da177e4SLinus Torvalds 23731da177e4SLinus Torvalds if (old_state != TCP_CLOSE) 23741da177e4SLinus Torvalds tcp_set_state(sk, TCP_CLOSE); 23751da177e4SLinus Torvalds 23761da177e4SLinus Torvalds /* ABORT function of RFC793 */ 23771da177e4SLinus Torvalds if (old_state == TCP_LISTEN) { 23780a5578cfSArnaldo Carvalho de Melo inet_csk_listen_stop(sk); 2379ee995283SPavel Emelyanov } else if (unlikely(tp->repair)) { 2380ee995283SPavel Emelyanov sk->sk_err = ECONNABORTED; 23811da177e4SLinus Torvalds } else if (tcp_need_reset(old_state) || 23821da177e4SLinus Torvalds (tp->snd_nxt != tp->write_seq && 23831da177e4SLinus Torvalds (1 << old_state) & (TCPF_CLOSING | TCPF_LAST_ACK))) { 2384caa20d9aSStephen Hemminger /* The last check adjusts for discrepancy of Linux wrt. RFC 23851da177e4SLinus Torvalds * states 23861da177e4SLinus Torvalds */ 23871da177e4SLinus Torvalds tcp_send_active_reset(sk, gfp_any()); 23881da177e4SLinus Torvalds sk->sk_err = ECONNRESET; 23891da177e4SLinus Torvalds } else if (old_state == TCP_SYN_SENT) 23901da177e4SLinus Torvalds sk->sk_err = ECONNRESET; 23911da177e4SLinus Torvalds 23921da177e4SLinus Torvalds tcp_clear_xmit_timers(sk); 23931da177e4SLinus Torvalds __skb_queue_purge(&sk->sk_receive_queue); 2394fe067e8aSDavid S. Miller tcp_write_queue_purge(sk); 2395cf1ef3f0SWei Wang tcp_fastopen_active_disable_ofo_check(sk); 23969f5afeaeSYaogong Wang skb_rbtree_purge(&tp->out_of_order_queue); 23971da177e4SLinus Torvalds 2398c720c7e8SEric Dumazet inet->inet_dport = 0; 23991da177e4SLinus Torvalds 24001da177e4SLinus Torvalds if (!(sk->sk_userlocks & SOCK_BINDADDR_LOCK)) 24011da177e4SLinus Torvalds inet_reset_saddr(sk); 24021da177e4SLinus Torvalds 24031da177e4SLinus Torvalds sk->sk_shutdown = 0; 24041da177e4SLinus Torvalds sock_reset_flag(sk, SOCK_DONE); 2405740b0f18SEric Dumazet tp->srtt_us = 0; 2406686a5624SYuvaraja Mariappan tp->write_seq += tp->max_window + 2; 2407686a5624SYuvaraja Mariappan if (tp->write_seq == 0) 24081da177e4SLinus Torvalds tp->write_seq = 1; 2409463c84b9SArnaldo Carvalho de Melo icsk->icsk_backoff = 0; 24101da177e4SLinus Torvalds tp->snd_cwnd = 2; 24116687e988SArnaldo Carvalho de Melo icsk->icsk_probes_out = 0; 24121da177e4SLinus Torvalds tp->packets_out = 0; 24130b6a05c1SIlpo Järvinen tp->snd_ssthresh = TCP_INFINITE_SSTHRESH; 24141da177e4SLinus Torvalds tp->snd_cwnd_cnt = 0; 24151fdf475aSEric Dumazet tp->window_clamp = 0; 24166687e988SArnaldo Carvalho de Melo tcp_set_ca_state(sk, TCP_CA_Open); 24171da177e4SLinus Torvalds tcp_clear_retrans(tp); 2418463c84b9SArnaldo Carvalho de Melo inet_csk_delack_init(sk); 2419499350a5SWei Wang /* Initialize rcv_mss to TCP_MIN_MSS to avoid division by 0 2420499350a5SWei Wang * issue in __tcp_select_window() 2421499350a5SWei Wang */ 2422499350a5SWei Wang icsk->icsk_ack.rcv_mss = TCP_MIN_MSS; 2423b40b4f79SSrinivas Aji memset(&tp->rx_opt, 0, sizeof(tp->rx_opt)); 24241da177e4SLinus Torvalds __sk_dst_reset(sk); 2425d747a7a5SWANG Cong dst_release(sk->sk_rx_dst); 2426d747a7a5SWANG Cong sk->sk_rx_dst = NULL; 242717c3060bSEric Dumazet tcp_saved_syn_free(tp); 24281da177e4SLinus Torvalds 24297db92362SWei Wang /* Clean up fastopen related fields */ 24307db92362SWei Wang tcp_free_fastopen_req(tp); 24317db92362SWei Wang inet->defer_connect = 0; 24327db92362SWei Wang 2433c720c7e8SEric Dumazet WARN_ON(inet->inet_num && !icsk->icsk_bind_hash); 24341da177e4SLinus Torvalds 24351da177e4SLinus Torvalds sk->sk_error_report(sk); 24361da177e4SLinus Torvalds return err; 24371da177e4SLinus Torvalds } 24384bc2f18bSEric Dumazet EXPORT_SYMBOL(tcp_disconnect); 24391da177e4SLinus Torvalds 2440a2a385d6SEric Dumazet static inline bool tcp_can_repair_sock(const struct sock *sk) 2441ee995283SPavel Emelyanov { 244252e804c6SEric W. Biederman return ns_capable(sock_net(sk)->user_ns, CAP_NET_ADMIN) && 2443319b0534SAndrey Vagin (sk->sk_state != TCP_LISTEN); 2444ee995283SPavel Emelyanov } 2445ee995283SPavel Emelyanov 2446b1ed4c4fSAndrey Vagin static int tcp_repair_set_window(struct tcp_sock *tp, char __user *optbuf, int len) 2447b1ed4c4fSAndrey Vagin { 2448b1ed4c4fSAndrey Vagin struct tcp_repair_window opt; 2449b1ed4c4fSAndrey Vagin 2450b1ed4c4fSAndrey Vagin if (!tp->repair) 2451b1ed4c4fSAndrey Vagin return -EPERM; 2452b1ed4c4fSAndrey Vagin 2453b1ed4c4fSAndrey Vagin if (len != sizeof(opt)) 2454b1ed4c4fSAndrey Vagin return -EINVAL; 2455b1ed4c4fSAndrey Vagin 2456b1ed4c4fSAndrey Vagin if (copy_from_user(&opt, optbuf, sizeof(opt))) 2457b1ed4c4fSAndrey Vagin return -EFAULT; 2458b1ed4c4fSAndrey Vagin 2459b1ed4c4fSAndrey Vagin if (opt.max_window < opt.snd_wnd) 2460b1ed4c4fSAndrey Vagin return -EINVAL; 2461b1ed4c4fSAndrey Vagin 2462b1ed4c4fSAndrey Vagin if (after(opt.snd_wl1, tp->rcv_nxt + opt.rcv_wnd)) 2463b1ed4c4fSAndrey Vagin return -EINVAL; 2464b1ed4c4fSAndrey Vagin 2465b1ed4c4fSAndrey Vagin if (after(opt.rcv_wup, tp->rcv_nxt)) 2466b1ed4c4fSAndrey Vagin return -EINVAL; 2467b1ed4c4fSAndrey Vagin 2468b1ed4c4fSAndrey Vagin tp->snd_wl1 = opt.snd_wl1; 2469b1ed4c4fSAndrey Vagin tp->snd_wnd = opt.snd_wnd; 2470b1ed4c4fSAndrey Vagin tp->max_window = opt.max_window; 2471b1ed4c4fSAndrey Vagin 2472b1ed4c4fSAndrey Vagin tp->rcv_wnd = opt.rcv_wnd; 2473b1ed4c4fSAndrey Vagin tp->rcv_wup = opt.rcv_wup; 2474b1ed4c4fSAndrey Vagin 2475b1ed4c4fSAndrey Vagin return 0; 2476b1ed4c4fSAndrey Vagin } 2477b1ed4c4fSAndrey Vagin 247815e56515SDouglas Caetano dos Santos static int tcp_repair_options_est(struct sock *sk, 2479de248a75SPavel Emelyanov struct tcp_repair_opt __user *optbuf, unsigned int len) 2480b139ba4eSPavel Emelyanov { 248115e56515SDouglas Caetano dos Santos struct tcp_sock *tp = tcp_sk(sk); 2482de248a75SPavel Emelyanov struct tcp_repair_opt opt; 2483b139ba4eSPavel Emelyanov 2484de248a75SPavel Emelyanov while (len >= sizeof(opt)) { 2485de248a75SPavel Emelyanov if (copy_from_user(&opt, optbuf, sizeof(opt))) 2486b139ba4eSPavel Emelyanov return -EFAULT; 2487b139ba4eSPavel Emelyanov 2488b139ba4eSPavel Emelyanov optbuf++; 2489de248a75SPavel Emelyanov len -= sizeof(opt); 2490b139ba4eSPavel Emelyanov 2491de248a75SPavel Emelyanov switch (opt.opt_code) { 2492de248a75SPavel Emelyanov case TCPOPT_MSS: 2493de248a75SPavel Emelyanov tp->rx_opt.mss_clamp = opt.opt_val; 249415e56515SDouglas Caetano dos Santos tcp_mtup_init(sk); 2495b139ba4eSPavel Emelyanov break; 2496de248a75SPavel Emelyanov case TCPOPT_WINDOW: 2497bc26ccd8SAndrey Vagin { 2498bc26ccd8SAndrey Vagin u16 snd_wscale = opt.opt_val & 0xFFFF; 2499bc26ccd8SAndrey Vagin u16 rcv_wscale = opt.opt_val >> 16; 2500bc26ccd8SAndrey Vagin 2501589c49cbSGao Feng if (snd_wscale > TCP_MAX_WSCALE || rcv_wscale > TCP_MAX_WSCALE) 2502b139ba4eSPavel Emelyanov return -EFBIG; 2503b139ba4eSPavel Emelyanov 2504bc26ccd8SAndrey Vagin tp->rx_opt.snd_wscale = snd_wscale; 2505bc26ccd8SAndrey Vagin tp->rx_opt.rcv_wscale = rcv_wscale; 2506bc26ccd8SAndrey Vagin tp->rx_opt.wscale_ok = 1; 2507bc26ccd8SAndrey Vagin } 2508b139ba4eSPavel Emelyanov break; 2509b139ba4eSPavel Emelyanov case TCPOPT_SACK_PERM: 2510de248a75SPavel Emelyanov if (opt.opt_val != 0) 2511de248a75SPavel Emelyanov return -EINVAL; 2512de248a75SPavel Emelyanov 2513b139ba4eSPavel Emelyanov tp->rx_opt.sack_ok |= TCP_SACK_SEEN; 2514b139ba4eSPavel Emelyanov if (sysctl_tcp_fack) 2515b139ba4eSPavel Emelyanov tcp_enable_fack(tp); 2516b139ba4eSPavel Emelyanov break; 2517b139ba4eSPavel Emelyanov case TCPOPT_TIMESTAMP: 2518de248a75SPavel Emelyanov if (opt.opt_val != 0) 2519de248a75SPavel Emelyanov return -EINVAL; 2520de248a75SPavel Emelyanov 2521b139ba4eSPavel Emelyanov tp->rx_opt.tstamp_ok = 1; 2522b139ba4eSPavel Emelyanov break; 2523b139ba4eSPavel Emelyanov } 2524b139ba4eSPavel Emelyanov } 2525b139ba4eSPavel Emelyanov 2526b139ba4eSPavel Emelyanov return 0; 2527b139ba4eSPavel Emelyanov } 2528b139ba4eSPavel Emelyanov 25291da177e4SLinus Torvalds /* 25301da177e4SLinus Torvalds * Socket option code for TCP. 25311da177e4SLinus Torvalds */ 25323fdadf7dSDmitry Mishin static int do_tcp_setsockopt(struct sock *sk, int level, 2533b7058842SDavid S. Miller int optname, char __user *optval, unsigned int optlen) 25341da177e4SLinus Torvalds { 25351da177e4SLinus Torvalds struct tcp_sock *tp = tcp_sk(sk); 2536463c84b9SArnaldo Carvalho de Melo struct inet_connection_sock *icsk = inet_csk(sk); 25371e579caaSNikolay Borisov struct net *net = sock_net(sk); 25381da177e4SLinus Torvalds int val; 25391da177e4SLinus Torvalds int err = 0; 25401da177e4SLinus Torvalds 2541e56fb50fSWilliam Allen Simpson /* These are data/string values, all the others are ints */ 2542e56fb50fSWilliam Allen Simpson switch (optname) { 2543e56fb50fSWilliam Allen Simpson case TCP_CONGESTION: { 25445f8ef48dSStephen Hemminger char name[TCP_CA_NAME_MAX]; 25455f8ef48dSStephen Hemminger 25465f8ef48dSStephen Hemminger if (optlen < 1) 25475f8ef48dSStephen Hemminger return -EINVAL; 25485f8ef48dSStephen Hemminger 25495f8ef48dSStephen Hemminger val = strncpy_from_user(name, optval, 25504fdb78d3SAndrew Morton min_t(long, TCP_CA_NAME_MAX-1, optlen)); 25515f8ef48dSStephen Hemminger if (val < 0) 25525f8ef48dSStephen Hemminger return -EFAULT; 25535f8ef48dSStephen Hemminger name[val] = 0; 25545f8ef48dSStephen Hemminger 25555f8ef48dSStephen Hemminger lock_sock(sk); 2556ebfa00c5SSabrina Dubroca err = tcp_set_congestion_control(sk, name, true, true); 25575f8ef48dSStephen Hemminger release_sock(sk); 25585f8ef48dSStephen Hemminger return err; 25595f8ef48dSStephen Hemminger } 2560734942ccSDave Watson case TCP_ULP: { 2561734942ccSDave Watson char name[TCP_ULP_NAME_MAX]; 2562734942ccSDave Watson 2563734942ccSDave Watson if (optlen < 1) 2564734942ccSDave Watson return -EINVAL; 2565734942ccSDave Watson 2566734942ccSDave Watson val = strncpy_from_user(name, optval, 2567734942ccSDave Watson min_t(long, TCP_ULP_NAME_MAX - 1, 2568734942ccSDave Watson optlen)); 2569734942ccSDave Watson if (val < 0) 2570734942ccSDave Watson return -EFAULT; 2571734942ccSDave Watson name[val] = 0; 2572734942ccSDave Watson 2573734942ccSDave Watson lock_sock(sk); 2574734942ccSDave Watson err = tcp_set_ulp(sk, name); 2575734942ccSDave Watson release_sock(sk); 2576734942ccSDave Watson return err; 2577734942ccSDave Watson } 25781fba70e5SYuchung Cheng case TCP_FASTOPEN_KEY: { 25791fba70e5SYuchung Cheng __u8 key[TCP_FASTOPEN_KEY_LENGTH]; 25801fba70e5SYuchung Cheng 25811fba70e5SYuchung Cheng if (optlen != sizeof(key)) 25821fba70e5SYuchung Cheng return -EINVAL; 25831fba70e5SYuchung Cheng 25841fba70e5SYuchung Cheng if (copy_from_user(key, optval, optlen)) 25851fba70e5SYuchung Cheng return -EFAULT; 25861fba70e5SYuchung Cheng 25871fba70e5SYuchung Cheng return tcp_fastopen_reset_cipher(net, sk, key, sizeof(key)); 25881fba70e5SYuchung Cheng } 2589e56fb50fSWilliam Allen Simpson default: 2590e56fb50fSWilliam Allen Simpson /* fallthru */ 2591e56fb50fSWilliam Allen Simpson break; 2592ccbd6a5aSJoe Perches } 25935f8ef48dSStephen Hemminger 25941da177e4SLinus Torvalds if (optlen < sizeof(int)) 25951da177e4SLinus Torvalds return -EINVAL; 25961da177e4SLinus Torvalds 25971da177e4SLinus Torvalds if (get_user(val, (int __user *)optval)) 25981da177e4SLinus Torvalds return -EFAULT; 25991da177e4SLinus Torvalds 26001da177e4SLinus Torvalds lock_sock(sk); 26011da177e4SLinus Torvalds 26021da177e4SLinus Torvalds switch (optname) { 26031da177e4SLinus Torvalds case TCP_MAXSEG: 26041da177e4SLinus Torvalds /* Values greater than interface MTU won't take effect. However 26051da177e4SLinus Torvalds * at the point when this call is done we typically don't yet 2606a777f715SRohit Chavan * know which interface is going to be used 2607a777f715SRohit Chavan */ 2608cfc62d87SGao Feng if (val && (val < TCP_MIN_MSS || val > MAX_TCP_WINDOW)) { 26091da177e4SLinus Torvalds err = -EINVAL; 26101da177e4SLinus Torvalds break; 26111da177e4SLinus Torvalds } 26121da177e4SLinus Torvalds tp->rx_opt.user_mss = val; 26131da177e4SLinus Torvalds break; 26141da177e4SLinus Torvalds 26151da177e4SLinus Torvalds case TCP_NODELAY: 26161da177e4SLinus Torvalds if (val) { 26171da177e4SLinus Torvalds /* TCP_NODELAY is weaker than TCP_CORK, so that 26181da177e4SLinus Torvalds * this option on corked socket is remembered, but 26191da177e4SLinus Torvalds * it is not activated until cork is cleared. 26201da177e4SLinus Torvalds * 26211da177e4SLinus Torvalds * However, when TCP_NODELAY is set we make 26221da177e4SLinus Torvalds * an explicit push, which overrides even TCP_CORK 26231da177e4SLinus Torvalds * for currently queued segments. 26241da177e4SLinus Torvalds */ 26251da177e4SLinus Torvalds tp->nonagle |= TCP_NAGLE_OFF|TCP_NAGLE_PUSH; 26269e412ba7SIlpo Järvinen tcp_push_pending_frames(sk); 26271da177e4SLinus Torvalds } else { 26281da177e4SLinus Torvalds tp->nonagle &= ~TCP_NAGLE_OFF; 26291da177e4SLinus Torvalds } 26301da177e4SLinus Torvalds break; 26311da177e4SLinus Torvalds 263236e31b0aSAndreas Petlund case TCP_THIN_LINEAR_TIMEOUTS: 263336e31b0aSAndreas Petlund if (val < 0 || val > 1) 263436e31b0aSAndreas Petlund err = -EINVAL; 263536e31b0aSAndreas Petlund else 263636e31b0aSAndreas Petlund tp->thin_lto = val; 263736e31b0aSAndreas Petlund break; 263836e31b0aSAndreas Petlund 26397e380175SAndreas Petlund case TCP_THIN_DUPACK: 26407e380175SAndreas Petlund if (val < 0 || val > 1) 26417e380175SAndreas Petlund err = -EINVAL; 26427e380175SAndreas Petlund break; 26437e380175SAndreas Petlund 2644ee995283SPavel Emelyanov case TCP_REPAIR: 2645ee995283SPavel Emelyanov if (!tcp_can_repair_sock(sk)) 2646ee995283SPavel Emelyanov err = -EPERM; 2647ee995283SPavel Emelyanov else if (val == 1) { 2648ee995283SPavel Emelyanov tp->repair = 1; 2649ee995283SPavel Emelyanov sk->sk_reuse = SK_FORCE_REUSE; 2650ee995283SPavel Emelyanov tp->repair_queue = TCP_NO_QUEUE; 2651ee995283SPavel Emelyanov } else if (val == 0) { 2652ee995283SPavel Emelyanov tp->repair = 0; 2653ee995283SPavel Emelyanov sk->sk_reuse = SK_NO_REUSE; 2654ee995283SPavel Emelyanov tcp_send_window_probe(sk); 2655ee995283SPavel Emelyanov } else 2656ee995283SPavel Emelyanov err = -EINVAL; 2657ee995283SPavel Emelyanov 2658ee995283SPavel Emelyanov break; 2659ee995283SPavel Emelyanov 2660ee995283SPavel Emelyanov case TCP_REPAIR_QUEUE: 2661ee995283SPavel Emelyanov if (!tp->repair) 2662ee995283SPavel Emelyanov err = -EPERM; 2663ee995283SPavel Emelyanov else if (val < TCP_QUEUES_NR) 2664ee995283SPavel Emelyanov tp->repair_queue = val; 2665ee995283SPavel Emelyanov else 2666ee995283SPavel Emelyanov err = -EINVAL; 2667ee995283SPavel Emelyanov break; 2668ee995283SPavel Emelyanov 2669ee995283SPavel Emelyanov case TCP_QUEUE_SEQ: 2670ee995283SPavel Emelyanov if (sk->sk_state != TCP_CLOSE) 2671ee995283SPavel Emelyanov err = -EPERM; 2672ee995283SPavel Emelyanov else if (tp->repair_queue == TCP_SEND_QUEUE) 2673ee995283SPavel Emelyanov tp->write_seq = val; 2674ee995283SPavel Emelyanov else if (tp->repair_queue == TCP_RECV_QUEUE) 2675ee995283SPavel Emelyanov tp->rcv_nxt = val; 2676ee995283SPavel Emelyanov else 2677ee995283SPavel Emelyanov err = -EINVAL; 2678ee995283SPavel Emelyanov break; 2679ee995283SPavel Emelyanov 2680b139ba4eSPavel Emelyanov case TCP_REPAIR_OPTIONS: 2681b139ba4eSPavel Emelyanov if (!tp->repair) 2682b139ba4eSPavel Emelyanov err = -EINVAL; 2683b139ba4eSPavel Emelyanov else if (sk->sk_state == TCP_ESTABLISHED) 268415e56515SDouglas Caetano dos Santos err = tcp_repair_options_est(sk, 2685de248a75SPavel Emelyanov (struct tcp_repair_opt __user *)optval, 2686de248a75SPavel Emelyanov optlen); 2687b139ba4eSPavel Emelyanov else 2688b139ba4eSPavel Emelyanov err = -EPERM; 2689b139ba4eSPavel Emelyanov break; 2690b139ba4eSPavel Emelyanov 26911da177e4SLinus Torvalds case TCP_CORK: 26921da177e4SLinus Torvalds /* When set indicates to always queue non-full frames. 26931da177e4SLinus Torvalds * Later the user clears this option and we transmit 26941da177e4SLinus Torvalds * any pending partial frames in the queue. This is 26951da177e4SLinus Torvalds * meant to be used alongside sendfile() to get properly 26961da177e4SLinus Torvalds * filled frames when the user (for example) must write 26971da177e4SLinus Torvalds * out headers with a write() call first and then use 26981da177e4SLinus Torvalds * sendfile to send out the data parts. 26991da177e4SLinus Torvalds * 27001da177e4SLinus Torvalds * TCP_CORK can be set together with TCP_NODELAY and it is 27011da177e4SLinus Torvalds * stronger than TCP_NODELAY. 27021da177e4SLinus Torvalds */ 27031da177e4SLinus Torvalds if (val) { 27041da177e4SLinus Torvalds tp->nonagle |= TCP_NAGLE_CORK; 27051da177e4SLinus Torvalds } else { 27061da177e4SLinus Torvalds tp->nonagle &= ~TCP_NAGLE_CORK; 27071da177e4SLinus Torvalds if (tp->nonagle&TCP_NAGLE_OFF) 27081da177e4SLinus Torvalds tp->nonagle |= TCP_NAGLE_PUSH; 27099e412ba7SIlpo Järvinen tcp_push_pending_frames(sk); 27101da177e4SLinus Torvalds } 27111da177e4SLinus Torvalds break; 27121da177e4SLinus Torvalds 27131da177e4SLinus Torvalds case TCP_KEEPIDLE: 27141da177e4SLinus Torvalds if (val < 1 || val > MAX_TCP_KEEPIDLE) 27151da177e4SLinus Torvalds err = -EINVAL; 27161da177e4SLinus Torvalds else { 27171da177e4SLinus Torvalds tp->keepalive_time = val * HZ; 27181da177e4SLinus Torvalds if (sock_flag(sk, SOCK_KEEPOPEN) && 27191da177e4SLinus Torvalds !((1 << sk->sk_state) & 27201da177e4SLinus Torvalds (TCPF_CLOSE | TCPF_LISTEN))) { 27216c37e5deSFlavio Leitner u32 elapsed = keepalive_time_elapsed(tp); 27221da177e4SLinus Torvalds if (tp->keepalive_time > elapsed) 27231da177e4SLinus Torvalds elapsed = tp->keepalive_time - elapsed; 27241da177e4SLinus Torvalds else 27251da177e4SLinus Torvalds elapsed = 0; 2726463c84b9SArnaldo Carvalho de Melo inet_csk_reset_keepalive_timer(sk, elapsed); 27271da177e4SLinus Torvalds } 27281da177e4SLinus Torvalds } 27291da177e4SLinus Torvalds break; 27301da177e4SLinus Torvalds case TCP_KEEPINTVL: 27311da177e4SLinus Torvalds if (val < 1 || val > MAX_TCP_KEEPINTVL) 27321da177e4SLinus Torvalds err = -EINVAL; 27331da177e4SLinus Torvalds else 27341da177e4SLinus Torvalds tp->keepalive_intvl = val * HZ; 27351da177e4SLinus Torvalds break; 27361da177e4SLinus Torvalds case TCP_KEEPCNT: 27371da177e4SLinus Torvalds if (val < 1 || val > MAX_TCP_KEEPCNT) 27381da177e4SLinus Torvalds err = -EINVAL; 27391da177e4SLinus Torvalds else 27401da177e4SLinus Torvalds tp->keepalive_probes = val; 27411da177e4SLinus Torvalds break; 27421da177e4SLinus Torvalds case TCP_SYNCNT: 27431da177e4SLinus Torvalds if (val < 1 || val > MAX_TCP_SYNCNT) 27441da177e4SLinus Torvalds err = -EINVAL; 27451da177e4SLinus Torvalds else 2746463c84b9SArnaldo Carvalho de Melo icsk->icsk_syn_retries = val; 27471da177e4SLinus Torvalds break; 27481da177e4SLinus Torvalds 2749cd8ae852SEric Dumazet case TCP_SAVE_SYN: 2750cd8ae852SEric Dumazet if (val < 0 || val > 1) 2751cd8ae852SEric Dumazet err = -EINVAL; 2752cd8ae852SEric Dumazet else 2753cd8ae852SEric Dumazet tp->save_syn = val; 2754cd8ae852SEric Dumazet break; 2755cd8ae852SEric Dumazet 27561da177e4SLinus Torvalds case TCP_LINGER2: 27571da177e4SLinus Torvalds if (val < 0) 27581da177e4SLinus Torvalds tp->linger2 = -1; 27591e579caaSNikolay Borisov else if (val > net->ipv4.sysctl_tcp_fin_timeout / HZ) 27601da177e4SLinus Torvalds tp->linger2 = 0; 27611da177e4SLinus Torvalds else 27621da177e4SLinus Torvalds tp->linger2 = val * HZ; 27631da177e4SLinus Torvalds break; 27641da177e4SLinus Torvalds 27651da177e4SLinus Torvalds case TCP_DEFER_ACCEPT: 2766b103cf34SJulian Anastasov /* Translate value in seconds to number of retransmits */ 2767b103cf34SJulian Anastasov icsk->icsk_accept_queue.rskq_defer_accept = 2768b103cf34SJulian Anastasov secs_to_retrans(val, TCP_TIMEOUT_INIT / HZ, 2769b103cf34SJulian Anastasov TCP_RTO_MAX / HZ); 27701da177e4SLinus Torvalds break; 27711da177e4SLinus Torvalds 27721da177e4SLinus Torvalds case TCP_WINDOW_CLAMP: 27731da177e4SLinus Torvalds if (!val) { 27741da177e4SLinus Torvalds if (sk->sk_state != TCP_CLOSE) { 27751da177e4SLinus Torvalds err = -EINVAL; 27761da177e4SLinus Torvalds break; 27771da177e4SLinus Torvalds } 27781da177e4SLinus Torvalds tp->window_clamp = 0; 27791da177e4SLinus Torvalds } else 27801da177e4SLinus Torvalds tp->window_clamp = val < SOCK_MIN_RCVBUF / 2 ? 27811da177e4SLinus Torvalds SOCK_MIN_RCVBUF / 2 : val; 27821da177e4SLinus Torvalds break; 27831da177e4SLinus Torvalds 27841da177e4SLinus Torvalds case TCP_QUICKACK: 27851da177e4SLinus Torvalds if (!val) { 2786463c84b9SArnaldo Carvalho de Melo icsk->icsk_ack.pingpong = 1; 27871da177e4SLinus Torvalds } else { 2788463c84b9SArnaldo Carvalho de Melo icsk->icsk_ack.pingpong = 0; 27891da177e4SLinus Torvalds if ((1 << sk->sk_state) & 27901da177e4SLinus Torvalds (TCPF_ESTABLISHED | TCPF_CLOSE_WAIT) && 2791463c84b9SArnaldo Carvalho de Melo inet_csk_ack_scheduled(sk)) { 2792463c84b9SArnaldo Carvalho de Melo icsk->icsk_ack.pending |= ICSK_ACK_PUSHED; 27930e4b4992SChris Leech tcp_cleanup_rbuf(sk, 1); 27941da177e4SLinus Torvalds if (!(val & 1)) 2795463c84b9SArnaldo Carvalho de Melo icsk->icsk_ack.pingpong = 1; 27961da177e4SLinus Torvalds } 27971da177e4SLinus Torvalds } 27981da177e4SLinus Torvalds break; 27991da177e4SLinus Torvalds 2800cfb6eeb4SYOSHIFUJI Hideaki #ifdef CONFIG_TCP_MD5SIG 2801cfb6eeb4SYOSHIFUJI Hideaki case TCP_MD5SIG: 28028917a777SIvan Delalande case TCP_MD5SIG_EXT: 2803cfb6eeb4SYOSHIFUJI Hideaki /* Read the IP->Key mappings from userspace */ 28048917a777SIvan Delalande err = tp->af_specific->md5_parse(sk, optname, optval, optlen); 2805cfb6eeb4SYOSHIFUJI Hideaki break; 2806cfb6eeb4SYOSHIFUJI Hideaki #endif 2807dca43c75SJerry Chu case TCP_USER_TIMEOUT: 2808b248230cSYuchung Cheng /* Cap the max time in ms TCP will retry or probe the window 2809dca43c75SJerry Chu * before giving up and aborting (ETIMEDOUT) a connection. 2810dca43c75SJerry Chu */ 281142493570SHangbin Liu if (val < 0) 281242493570SHangbin Liu err = -EINVAL; 281342493570SHangbin Liu else 2814dca43c75SJerry Chu icsk->icsk_user_timeout = msecs_to_jiffies(val); 2815dca43c75SJerry Chu break; 28168336886fSJerry Chu 28178336886fSJerry Chu case TCP_FASTOPEN: 28188336886fSJerry Chu if (val >= 0 && ((1 << sk->sk_state) & (TCPF_CLOSE | 2819dfea2aa6SChristoph Paasch TCPF_LISTEN))) { 282043713848SHaishuang Yan tcp_fastopen_init_key_once(net); 2821dfea2aa6SChristoph Paasch 28220536fcc0SEric Dumazet fastopen_queue_tune(sk, val); 2823dfea2aa6SChristoph Paasch } else { 28248336886fSJerry Chu err = -EINVAL; 2825dfea2aa6SChristoph Paasch } 28268336886fSJerry Chu break; 282719f6d3f3SWei Wang case TCP_FASTOPEN_CONNECT: 282819f6d3f3SWei Wang if (val > 1 || val < 0) { 282919f6d3f3SWei Wang err = -EINVAL; 2830e1cfcbe8SHaishuang Yan } else if (net->ipv4.sysctl_tcp_fastopen & TFO_CLIENT_ENABLE) { 283119f6d3f3SWei Wang if (sk->sk_state == TCP_CLOSE) 283219f6d3f3SWei Wang tp->fastopen_connect = val; 283319f6d3f3SWei Wang else 283419f6d3f3SWei Wang err = -EINVAL; 283519f6d3f3SWei Wang } else { 283619f6d3f3SWei Wang err = -EOPNOTSUPP; 283719f6d3f3SWei Wang } 283819f6d3f3SWei Wang break; 2839*71c02379SChristoph Paasch case TCP_FASTOPEN_NO_COOKIE: 2840*71c02379SChristoph Paasch if (val > 1 || val < 0) 2841*71c02379SChristoph Paasch err = -EINVAL; 2842*71c02379SChristoph Paasch else if (!((1 << sk->sk_state) & (TCPF_CLOSE | TCPF_LISTEN))) 2843*71c02379SChristoph Paasch err = -EINVAL; 2844*71c02379SChristoph Paasch else 2845*71c02379SChristoph Paasch tp->fastopen_no_cookie = val; 2846*71c02379SChristoph Paasch break; 284793be6ce0SAndrey Vagin case TCP_TIMESTAMP: 284893be6ce0SAndrey Vagin if (!tp->repair) 284993be6ce0SAndrey Vagin err = -EPERM; 285093be6ce0SAndrey Vagin else 28519a568de4SEric Dumazet tp->tsoffset = val - tcp_time_stamp_raw(); 285293be6ce0SAndrey Vagin break; 2853b1ed4c4fSAndrey Vagin case TCP_REPAIR_WINDOW: 2854b1ed4c4fSAndrey Vagin err = tcp_repair_set_window(tp, optval, optlen); 2855b1ed4c4fSAndrey Vagin break; 2856c9bee3b7SEric Dumazet case TCP_NOTSENT_LOWAT: 2857c9bee3b7SEric Dumazet tp->notsent_lowat = val; 2858c9bee3b7SEric Dumazet sk->sk_write_space(sk); 2859c9bee3b7SEric Dumazet break; 28601da177e4SLinus Torvalds default: 28611da177e4SLinus Torvalds err = -ENOPROTOOPT; 28621da177e4SLinus Torvalds break; 28633ff50b79SStephen Hemminger } 28643ff50b79SStephen Hemminger 28651da177e4SLinus Torvalds release_sock(sk); 28661da177e4SLinus Torvalds return err; 28671da177e4SLinus Torvalds } 28681da177e4SLinus Torvalds 28693fdadf7dSDmitry Mishin int tcp_setsockopt(struct sock *sk, int level, int optname, char __user *optval, 2870b7058842SDavid S. Miller unsigned int optlen) 28713fdadf7dSDmitry Mishin { 2872cf533ea5SEric Dumazet const struct inet_connection_sock *icsk = inet_csk(sk); 28733fdadf7dSDmitry Mishin 28743fdadf7dSDmitry Mishin if (level != SOL_TCP) 28753fdadf7dSDmitry Mishin return icsk->icsk_af_ops->setsockopt(sk, level, optname, 28763fdadf7dSDmitry Mishin optval, optlen); 28773fdadf7dSDmitry Mishin return do_tcp_setsockopt(sk, level, optname, optval, optlen); 28783fdadf7dSDmitry Mishin } 28794bc2f18bSEric Dumazet EXPORT_SYMBOL(tcp_setsockopt); 28803fdadf7dSDmitry Mishin 28813fdadf7dSDmitry Mishin #ifdef CONFIG_COMPAT 2882543d9cfeSArnaldo Carvalho de Melo int compat_tcp_setsockopt(struct sock *sk, int level, int optname, 2883b7058842SDavid S. Miller char __user *optval, unsigned int optlen) 28843fdadf7dSDmitry Mishin { 2885dec73ff0SArnaldo Carvalho de Melo if (level != SOL_TCP) 2886dec73ff0SArnaldo Carvalho de Melo return inet_csk_compat_setsockopt(sk, level, optname, 2887dec73ff0SArnaldo Carvalho de Melo optval, optlen); 28883fdadf7dSDmitry Mishin return do_tcp_setsockopt(sk, level, optname, optval, optlen); 28893fdadf7dSDmitry Mishin } 2890543d9cfeSArnaldo Carvalho de Melo EXPORT_SYMBOL(compat_tcp_setsockopt); 28913fdadf7dSDmitry Mishin #endif 28923fdadf7dSDmitry Mishin 2893efd90174SFrancis Yan static void tcp_get_info_chrono_stats(const struct tcp_sock *tp, 2894efd90174SFrancis Yan struct tcp_info *info) 2895efd90174SFrancis Yan { 2896efd90174SFrancis Yan u64 stats[__TCP_CHRONO_MAX], total = 0; 2897efd90174SFrancis Yan enum tcp_chrono i; 2898efd90174SFrancis Yan 2899efd90174SFrancis Yan for (i = TCP_CHRONO_BUSY; i < __TCP_CHRONO_MAX; ++i) { 2900efd90174SFrancis Yan stats[i] = tp->chrono_stat[i - 1]; 2901efd90174SFrancis Yan if (i == tp->chrono_type) 2902628174ccSEric Dumazet stats[i] += tcp_jiffies32 - tp->chrono_start; 2903efd90174SFrancis Yan stats[i] *= USEC_PER_SEC / HZ; 2904efd90174SFrancis Yan total += stats[i]; 2905efd90174SFrancis Yan } 2906efd90174SFrancis Yan 2907efd90174SFrancis Yan info->tcpi_busy_time = total; 2908efd90174SFrancis Yan info->tcpi_rwnd_limited = stats[TCP_CHRONO_RWND_LIMITED]; 2909efd90174SFrancis Yan info->tcpi_sndbuf_limited = stats[TCP_CHRONO_SNDBUF_LIMITED]; 2910efd90174SFrancis Yan } 2911efd90174SFrancis Yan 29121da177e4SLinus Torvalds /* Return information about state of tcp endpoint in API format. */ 29130df48c26SEric Dumazet void tcp_get_info(struct sock *sk, struct tcp_info *info) 29141da177e4SLinus Torvalds { 291535ac838aSCraig Gallek const struct tcp_sock *tp = tcp_sk(sk); /* iff sk_type == SOCK_STREAM */ 2916463c84b9SArnaldo Carvalho de Melo const struct inet_connection_sock *icsk = inet_csk(sk); 29170263598cSWei Wang u32 now; 2918ff5d7497SEric Dumazet u64 rate64; 291967db3e4bSEric Dumazet bool slow; 2920fad9dfefSEric Dumazet u32 rate; 29211da177e4SLinus Torvalds 29221da177e4SLinus Torvalds memset(info, 0, sizeof(*info)); 292335ac838aSCraig Gallek if (sk->sk_type != SOCK_STREAM) 292435ac838aSCraig Gallek return; 29251da177e4SLinus Torvalds 292600fd38d9SEric Dumazet info->tcpi_state = sk_state_load(sk); 292700fd38d9SEric Dumazet 2928ccbf3bfaSEric Dumazet /* Report meaningful fields for all TCP states, including listeners */ 2929ccbf3bfaSEric Dumazet rate = READ_ONCE(sk->sk_pacing_rate); 2930ccbf3bfaSEric Dumazet rate64 = rate != ~0U ? rate : ~0ULL; 2931f522a5fcSEric Dumazet info->tcpi_pacing_rate = rate64; 2932ccbf3bfaSEric Dumazet 2933ccbf3bfaSEric Dumazet rate = READ_ONCE(sk->sk_max_pacing_rate); 2934ccbf3bfaSEric Dumazet rate64 = rate != ~0U ? rate : ~0ULL; 2935f522a5fcSEric Dumazet info->tcpi_max_pacing_rate = rate64; 2936ccbf3bfaSEric Dumazet 2937ccbf3bfaSEric Dumazet info->tcpi_reordering = tp->reordering; 2938ccbf3bfaSEric Dumazet info->tcpi_snd_cwnd = tp->snd_cwnd; 2939ccbf3bfaSEric Dumazet 2940ccbf3bfaSEric Dumazet if (info->tcpi_state == TCP_LISTEN) { 2941ccbf3bfaSEric Dumazet /* listeners aliased fields : 2942ccbf3bfaSEric Dumazet * tcpi_unacked -> Number of children ready for accept() 2943ccbf3bfaSEric Dumazet * tcpi_sacked -> max backlog 2944ccbf3bfaSEric Dumazet */ 2945ccbf3bfaSEric Dumazet info->tcpi_unacked = sk->sk_ack_backlog; 2946ccbf3bfaSEric Dumazet info->tcpi_sacked = sk->sk_max_ack_backlog; 2947ccbf3bfaSEric Dumazet return; 2948ccbf3bfaSEric Dumazet } 2949b369e7fdSEric Dumazet 2950b369e7fdSEric Dumazet slow = lock_sock_fast(sk); 2951b369e7fdSEric Dumazet 29526687e988SArnaldo Carvalho de Melo info->tcpi_ca_state = icsk->icsk_ca_state; 2953463c84b9SArnaldo Carvalho de Melo info->tcpi_retransmits = icsk->icsk_retransmits; 29546687e988SArnaldo Carvalho de Melo info->tcpi_probes = icsk->icsk_probes_out; 2955463c84b9SArnaldo Carvalho de Melo info->tcpi_backoff = icsk->icsk_backoff; 29561da177e4SLinus Torvalds 29571da177e4SLinus Torvalds if (tp->rx_opt.tstamp_ok) 29581da177e4SLinus Torvalds info->tcpi_options |= TCPI_OPT_TIMESTAMPS; 2959e60402d0SIlpo Järvinen if (tcp_is_sack(tp)) 29601da177e4SLinus Torvalds info->tcpi_options |= TCPI_OPT_SACK; 29611da177e4SLinus Torvalds if (tp->rx_opt.wscale_ok) { 29621da177e4SLinus Torvalds info->tcpi_options |= TCPI_OPT_WSCALE; 29631da177e4SLinus Torvalds info->tcpi_snd_wscale = tp->rx_opt.snd_wscale; 29641da177e4SLinus Torvalds info->tcpi_rcv_wscale = tp->rx_opt.rcv_wscale; 29651da177e4SLinus Torvalds } 29661da177e4SLinus Torvalds 29671da177e4SLinus Torvalds if (tp->ecn_flags & TCP_ECN_OK) 29681da177e4SLinus Torvalds info->tcpi_options |= TCPI_OPT_ECN; 2969b5c5693bSEric Dumazet if (tp->ecn_flags & TCP_ECN_SEEN) 2970b5c5693bSEric Dumazet info->tcpi_options |= TCPI_OPT_ECN_SEEN; 29716f73601eSYuchung Cheng if (tp->syn_data_acked) 29726f73601eSYuchung Cheng info->tcpi_options |= TCPI_OPT_SYN_DATA; 29731da177e4SLinus Torvalds 2974463c84b9SArnaldo Carvalho de Melo info->tcpi_rto = jiffies_to_usecs(icsk->icsk_rto); 2975463c84b9SArnaldo Carvalho de Melo info->tcpi_ato = jiffies_to_usecs(icsk->icsk_ack.ato); 2976c1b4a7e6SDavid S. Miller info->tcpi_snd_mss = tp->mss_cache; 2977463c84b9SArnaldo Carvalho de Melo info->tcpi_rcv_mss = icsk->icsk_ack.rcv_mss; 29781da177e4SLinus Torvalds 29791da177e4SLinus Torvalds info->tcpi_unacked = tp->packets_out; 29801da177e4SLinus Torvalds info->tcpi_sacked = tp->sacked_out; 2981ccbf3bfaSEric Dumazet 29821da177e4SLinus Torvalds info->tcpi_lost = tp->lost_out; 29831da177e4SLinus Torvalds info->tcpi_retrans = tp->retrans_out; 29841da177e4SLinus Torvalds info->tcpi_fackets = tp->fackets_out; 29851da177e4SLinus Torvalds 2986d635fbe2SEric Dumazet now = tcp_jiffies32; 29871da177e4SLinus Torvalds info->tcpi_last_data_sent = jiffies_to_msecs(now - tp->lsndtime); 2988463c84b9SArnaldo Carvalho de Melo info->tcpi_last_data_recv = jiffies_to_msecs(now - icsk->icsk_ack.lrcvtime); 29891da177e4SLinus Torvalds info->tcpi_last_ack_recv = jiffies_to_msecs(now - tp->rcv_tstamp); 29901da177e4SLinus Torvalds 2991d83d8461SArnaldo Carvalho de Melo info->tcpi_pmtu = icsk->icsk_pmtu_cookie; 29921da177e4SLinus Torvalds info->tcpi_rcv_ssthresh = tp->rcv_ssthresh; 2993740b0f18SEric Dumazet info->tcpi_rtt = tp->srtt_us >> 3; 2994740b0f18SEric Dumazet info->tcpi_rttvar = tp->mdev_us >> 2; 29951da177e4SLinus Torvalds info->tcpi_snd_ssthresh = tp->snd_ssthresh; 29961da177e4SLinus Torvalds info->tcpi_advmss = tp->advmss; 29971da177e4SLinus Torvalds 2998645f4c6fSEric Dumazet info->tcpi_rcv_rtt = tp->rcv_rtt_est.rtt_us >> 3; 29991da177e4SLinus Torvalds info->tcpi_rcv_space = tp->rcvq_space.space; 30001da177e4SLinus Torvalds 30011da177e4SLinus Torvalds info->tcpi_total_retrans = tp->total_retrans; 3002977cb0ecSEric Dumazet 3003f522a5fcSEric Dumazet info->tcpi_bytes_acked = tp->bytes_acked; 3004f522a5fcSEric Dumazet info->tcpi_bytes_received = tp->bytes_received; 300567db3e4bSEric Dumazet info->tcpi_notsent_bytes = max_t(int, 0, tp->write_seq - tp->snd_nxt); 3006efd90174SFrancis Yan tcp_get_info_chrono_stats(tp, info); 300767db3e4bSEric Dumazet 30082efd055cSMarcelo Ricardo Leitner info->tcpi_segs_out = tp->segs_out; 30092efd055cSMarcelo Ricardo Leitner info->tcpi_segs_in = tp->segs_in; 3010cd9b2660SEric Dumazet 3011cd9b2660SEric Dumazet info->tcpi_min_rtt = tcp_min_rtt(tp); 3012a44d6eacSMartin KaFai Lau info->tcpi_data_segs_in = tp->data_segs_in; 3013a44d6eacSMartin KaFai Lau info->tcpi_data_segs_out = tp->data_segs_out; 3014eb8329e0SYuchung Cheng 3015eb8329e0SYuchung Cheng info->tcpi_delivery_rate_app_limited = tp->rate_app_limited ? 1 : 0; 30160263598cSWei Wang rate64 = tcp_compute_delivery_rate(tp); 30170263598cSWei Wang if (rate64) 3018f522a5fcSEric Dumazet info->tcpi_delivery_rate = rate64; 3019b369e7fdSEric Dumazet unlock_sock_fast(sk, slow); 30201da177e4SLinus Torvalds } 30211da177e4SLinus Torvalds EXPORT_SYMBOL_GPL(tcp_get_info); 30221da177e4SLinus Torvalds 30231c885808SFrancis Yan struct sk_buff *tcp_get_timestamping_opt_stats(const struct sock *sk) 30241c885808SFrancis Yan { 30251c885808SFrancis Yan const struct tcp_sock *tp = tcp_sk(sk); 30261c885808SFrancis Yan struct sk_buff *stats; 30271c885808SFrancis Yan struct tcp_info info; 3028bb7c19f9SWei Wang u64 rate64; 3029bb7c19f9SWei Wang u32 rate; 30301c885808SFrancis Yan 3031bb7c19f9SWei Wang stats = alloc_skb(7 * nla_total_size_64bit(sizeof(u64)) + 3032bb7c19f9SWei Wang 3 * nla_total_size(sizeof(u32)) + 3033bb7c19f9SWei Wang 2 * nla_total_size(sizeof(u8)), GFP_ATOMIC); 30341c885808SFrancis Yan if (!stats) 30351c885808SFrancis Yan return NULL; 30361c885808SFrancis Yan 30371c885808SFrancis Yan tcp_get_info_chrono_stats(tp, &info); 30381c885808SFrancis Yan nla_put_u64_64bit(stats, TCP_NLA_BUSY, 30391c885808SFrancis Yan info.tcpi_busy_time, TCP_NLA_PAD); 30401c885808SFrancis Yan nla_put_u64_64bit(stats, TCP_NLA_RWND_LIMITED, 30411c885808SFrancis Yan info.tcpi_rwnd_limited, TCP_NLA_PAD); 30421c885808SFrancis Yan nla_put_u64_64bit(stats, TCP_NLA_SNDBUF_LIMITED, 30431c885808SFrancis Yan info.tcpi_sndbuf_limited, TCP_NLA_PAD); 30447e98102fSYuchung Cheng nla_put_u64_64bit(stats, TCP_NLA_DATA_SEGS_OUT, 30457e98102fSYuchung Cheng tp->data_segs_out, TCP_NLA_PAD); 30467e98102fSYuchung Cheng nla_put_u64_64bit(stats, TCP_NLA_TOTAL_RETRANS, 30477e98102fSYuchung Cheng tp->total_retrans, TCP_NLA_PAD); 3048bb7c19f9SWei Wang 3049bb7c19f9SWei Wang rate = READ_ONCE(sk->sk_pacing_rate); 3050bb7c19f9SWei Wang rate64 = rate != ~0U ? rate : ~0ULL; 3051bb7c19f9SWei Wang nla_put_u64_64bit(stats, TCP_NLA_PACING_RATE, rate64, TCP_NLA_PAD); 3052bb7c19f9SWei Wang 3053bb7c19f9SWei Wang rate64 = tcp_compute_delivery_rate(tp); 3054bb7c19f9SWei Wang nla_put_u64_64bit(stats, TCP_NLA_DELIVERY_RATE, rate64, TCP_NLA_PAD); 3055bb7c19f9SWei Wang 3056bb7c19f9SWei Wang nla_put_u32(stats, TCP_NLA_SND_CWND, tp->snd_cwnd); 3057bb7c19f9SWei Wang nla_put_u32(stats, TCP_NLA_REORDERING, tp->reordering); 3058bb7c19f9SWei Wang nla_put_u32(stats, TCP_NLA_MIN_RTT, tcp_min_rtt(tp)); 3059bb7c19f9SWei Wang 3060bb7c19f9SWei Wang nla_put_u8(stats, TCP_NLA_RECUR_RETRANS, inet_csk(sk)->icsk_retransmits); 3061bb7c19f9SWei Wang nla_put_u8(stats, TCP_NLA_DELIVERY_RATE_APP_LMT, !!tp->rate_app_limited); 30621c885808SFrancis Yan return stats; 30631c885808SFrancis Yan } 30641c885808SFrancis Yan 30653fdadf7dSDmitry Mishin static int do_tcp_getsockopt(struct sock *sk, int level, 30663fdadf7dSDmitry Mishin int optname, char __user *optval, int __user *optlen) 30671da177e4SLinus Torvalds { 3068295f7324SArnaldo Carvalho de Melo struct inet_connection_sock *icsk = inet_csk(sk); 30691da177e4SLinus Torvalds struct tcp_sock *tp = tcp_sk(sk); 30706fa25166SNikolay Borisov struct net *net = sock_net(sk); 30711da177e4SLinus Torvalds int val, len; 30721da177e4SLinus Torvalds 30731da177e4SLinus Torvalds if (get_user(len, optlen)) 30741da177e4SLinus Torvalds return -EFAULT; 30751da177e4SLinus Torvalds 30761da177e4SLinus Torvalds len = min_t(unsigned int, len, sizeof(int)); 30771da177e4SLinus Torvalds 30781da177e4SLinus Torvalds if (len < 0) 30791da177e4SLinus Torvalds return -EINVAL; 30801da177e4SLinus Torvalds 30811da177e4SLinus Torvalds switch (optname) { 30821da177e4SLinus Torvalds case TCP_MAXSEG: 3083c1b4a7e6SDavid S. Miller val = tp->mss_cache; 30841da177e4SLinus Torvalds if (!val && ((1 << sk->sk_state) & (TCPF_CLOSE | TCPF_LISTEN))) 30851da177e4SLinus Torvalds val = tp->rx_opt.user_mss; 30865e6a3ce6SPavel Emelyanov if (tp->repair) 30875e6a3ce6SPavel Emelyanov val = tp->rx_opt.mss_clamp; 30881da177e4SLinus Torvalds break; 30891da177e4SLinus Torvalds case TCP_NODELAY: 30901da177e4SLinus Torvalds val = !!(tp->nonagle&TCP_NAGLE_OFF); 30911da177e4SLinus Torvalds break; 30921da177e4SLinus Torvalds case TCP_CORK: 30931da177e4SLinus Torvalds val = !!(tp->nonagle&TCP_NAGLE_CORK); 30941da177e4SLinus Torvalds break; 30951da177e4SLinus Torvalds case TCP_KEEPIDLE: 3096df19a626SEric Dumazet val = keepalive_time_when(tp) / HZ; 30971da177e4SLinus Torvalds break; 30981da177e4SLinus Torvalds case TCP_KEEPINTVL: 3099df19a626SEric Dumazet val = keepalive_intvl_when(tp) / HZ; 31001da177e4SLinus Torvalds break; 31011da177e4SLinus Torvalds case TCP_KEEPCNT: 3102df19a626SEric Dumazet val = keepalive_probes(tp); 31031da177e4SLinus Torvalds break; 31041da177e4SLinus Torvalds case TCP_SYNCNT: 31056fa25166SNikolay Borisov val = icsk->icsk_syn_retries ? : net->ipv4.sysctl_tcp_syn_retries; 31061da177e4SLinus Torvalds break; 31071da177e4SLinus Torvalds case TCP_LINGER2: 31081da177e4SLinus Torvalds val = tp->linger2; 31091da177e4SLinus Torvalds if (val >= 0) 31101e579caaSNikolay Borisov val = (val ? : net->ipv4.sysctl_tcp_fin_timeout) / HZ; 31111da177e4SLinus Torvalds break; 31121da177e4SLinus Torvalds case TCP_DEFER_ACCEPT: 3113b103cf34SJulian Anastasov val = retrans_to_secs(icsk->icsk_accept_queue.rskq_defer_accept, 3114b103cf34SJulian Anastasov TCP_TIMEOUT_INIT / HZ, TCP_RTO_MAX / HZ); 31151da177e4SLinus Torvalds break; 31161da177e4SLinus Torvalds case TCP_WINDOW_CLAMP: 31171da177e4SLinus Torvalds val = tp->window_clamp; 31181da177e4SLinus Torvalds break; 31191da177e4SLinus Torvalds case TCP_INFO: { 31201da177e4SLinus Torvalds struct tcp_info info; 31211da177e4SLinus Torvalds 31221da177e4SLinus Torvalds if (get_user(len, optlen)) 31231da177e4SLinus Torvalds return -EFAULT; 31241da177e4SLinus Torvalds 31251da177e4SLinus Torvalds tcp_get_info(sk, &info); 31261da177e4SLinus Torvalds 31271da177e4SLinus Torvalds len = min_t(unsigned int, len, sizeof(info)); 31281da177e4SLinus Torvalds if (put_user(len, optlen)) 31291da177e4SLinus Torvalds return -EFAULT; 31301da177e4SLinus Torvalds if (copy_to_user(optval, &info, len)) 31311da177e4SLinus Torvalds return -EFAULT; 31321da177e4SLinus Torvalds return 0; 31331da177e4SLinus Torvalds } 31346e9250f5SEric Dumazet case TCP_CC_INFO: { 31356e9250f5SEric Dumazet const struct tcp_congestion_ops *ca_ops; 31366e9250f5SEric Dumazet union tcp_cc_info info; 31376e9250f5SEric Dumazet size_t sz = 0; 31386e9250f5SEric Dumazet int attr; 31396e9250f5SEric Dumazet 31406e9250f5SEric Dumazet if (get_user(len, optlen)) 31416e9250f5SEric Dumazet return -EFAULT; 31426e9250f5SEric Dumazet 31436e9250f5SEric Dumazet ca_ops = icsk->icsk_ca_ops; 31446e9250f5SEric Dumazet if (ca_ops && ca_ops->get_info) 31456e9250f5SEric Dumazet sz = ca_ops->get_info(sk, ~0U, &attr, &info); 31466e9250f5SEric Dumazet 31476e9250f5SEric Dumazet len = min_t(unsigned int, len, sz); 31486e9250f5SEric Dumazet if (put_user(len, optlen)) 31496e9250f5SEric Dumazet return -EFAULT; 31506e9250f5SEric Dumazet if (copy_to_user(optval, &info, len)) 31516e9250f5SEric Dumazet return -EFAULT; 31526e9250f5SEric Dumazet return 0; 31536e9250f5SEric Dumazet } 31541da177e4SLinus Torvalds case TCP_QUICKACK: 3155295f7324SArnaldo Carvalho de Melo val = !icsk->icsk_ack.pingpong; 31561da177e4SLinus Torvalds break; 31575f8ef48dSStephen Hemminger 31585f8ef48dSStephen Hemminger case TCP_CONGESTION: 31595f8ef48dSStephen Hemminger if (get_user(len, optlen)) 31605f8ef48dSStephen Hemminger return -EFAULT; 31615f8ef48dSStephen Hemminger len = min_t(unsigned int, len, TCP_CA_NAME_MAX); 31625f8ef48dSStephen Hemminger if (put_user(len, optlen)) 31635f8ef48dSStephen Hemminger return -EFAULT; 31646687e988SArnaldo Carvalho de Melo if (copy_to_user(optval, icsk->icsk_ca_ops->name, len)) 31655f8ef48dSStephen Hemminger return -EFAULT; 31665f8ef48dSStephen Hemminger return 0; 3167e56fb50fSWilliam Allen Simpson 3168734942ccSDave Watson case TCP_ULP: 3169734942ccSDave Watson if (get_user(len, optlen)) 3170734942ccSDave Watson return -EFAULT; 3171734942ccSDave Watson len = min_t(unsigned int, len, TCP_ULP_NAME_MAX); 3172d97af30fSDave Watson if (!icsk->icsk_ulp_ops) { 3173d97af30fSDave Watson if (put_user(0, optlen)) 3174d97af30fSDave Watson return -EFAULT; 3175d97af30fSDave Watson return 0; 3176d97af30fSDave Watson } 3177734942ccSDave Watson if (put_user(len, optlen)) 3178734942ccSDave Watson return -EFAULT; 3179734942ccSDave Watson if (copy_to_user(optval, icsk->icsk_ulp_ops->name, len)) 3180734942ccSDave Watson return -EFAULT; 3181734942ccSDave Watson return 0; 3182734942ccSDave Watson 31831fba70e5SYuchung Cheng case TCP_FASTOPEN_KEY: { 31841fba70e5SYuchung Cheng __u8 key[TCP_FASTOPEN_KEY_LENGTH]; 31851fba70e5SYuchung Cheng struct tcp_fastopen_context *ctx; 31861fba70e5SYuchung Cheng 31871fba70e5SYuchung Cheng if (get_user(len, optlen)) 31881fba70e5SYuchung Cheng return -EFAULT; 31891fba70e5SYuchung Cheng 31901fba70e5SYuchung Cheng rcu_read_lock(); 31911fba70e5SYuchung Cheng ctx = rcu_dereference(icsk->icsk_accept_queue.fastopenq.ctx); 31921fba70e5SYuchung Cheng if (ctx) 31931fba70e5SYuchung Cheng memcpy(key, ctx->key, sizeof(key)); 31941fba70e5SYuchung Cheng else 31951fba70e5SYuchung Cheng len = 0; 31961fba70e5SYuchung Cheng rcu_read_unlock(); 31971fba70e5SYuchung Cheng 31981fba70e5SYuchung Cheng len = min_t(unsigned int, len, sizeof(key)); 31991fba70e5SYuchung Cheng if (put_user(len, optlen)) 32001fba70e5SYuchung Cheng return -EFAULT; 32011fba70e5SYuchung Cheng if (copy_to_user(optval, key, len)) 32021fba70e5SYuchung Cheng return -EFAULT; 32031fba70e5SYuchung Cheng return 0; 32041fba70e5SYuchung Cheng } 32053c0fef0bSJosh Hunt case TCP_THIN_LINEAR_TIMEOUTS: 32063c0fef0bSJosh Hunt val = tp->thin_lto; 32073c0fef0bSJosh Hunt break; 32084a7f6009SYuchung Cheng 32093c0fef0bSJosh Hunt case TCP_THIN_DUPACK: 32104a7f6009SYuchung Cheng val = 0; 32113c0fef0bSJosh Hunt break; 3212dca43c75SJerry Chu 3213ee995283SPavel Emelyanov case TCP_REPAIR: 3214ee995283SPavel Emelyanov val = tp->repair; 3215ee995283SPavel Emelyanov break; 3216ee995283SPavel Emelyanov 3217ee995283SPavel Emelyanov case TCP_REPAIR_QUEUE: 3218ee995283SPavel Emelyanov if (tp->repair) 3219ee995283SPavel Emelyanov val = tp->repair_queue; 3220ee995283SPavel Emelyanov else 3221ee995283SPavel Emelyanov return -EINVAL; 3222ee995283SPavel Emelyanov break; 3223ee995283SPavel Emelyanov 3224b1ed4c4fSAndrey Vagin case TCP_REPAIR_WINDOW: { 3225b1ed4c4fSAndrey Vagin struct tcp_repair_window opt; 3226b1ed4c4fSAndrey Vagin 3227b1ed4c4fSAndrey Vagin if (get_user(len, optlen)) 3228b1ed4c4fSAndrey Vagin return -EFAULT; 3229b1ed4c4fSAndrey Vagin 3230b1ed4c4fSAndrey Vagin if (len != sizeof(opt)) 3231b1ed4c4fSAndrey Vagin return -EINVAL; 3232b1ed4c4fSAndrey Vagin 3233b1ed4c4fSAndrey Vagin if (!tp->repair) 3234b1ed4c4fSAndrey Vagin return -EPERM; 3235b1ed4c4fSAndrey Vagin 3236b1ed4c4fSAndrey Vagin opt.snd_wl1 = tp->snd_wl1; 3237b1ed4c4fSAndrey Vagin opt.snd_wnd = tp->snd_wnd; 3238b1ed4c4fSAndrey Vagin opt.max_window = tp->max_window; 3239b1ed4c4fSAndrey Vagin opt.rcv_wnd = tp->rcv_wnd; 3240b1ed4c4fSAndrey Vagin opt.rcv_wup = tp->rcv_wup; 3241b1ed4c4fSAndrey Vagin 3242b1ed4c4fSAndrey Vagin if (copy_to_user(optval, &opt, len)) 3243b1ed4c4fSAndrey Vagin return -EFAULT; 3244b1ed4c4fSAndrey Vagin return 0; 3245b1ed4c4fSAndrey Vagin } 3246ee995283SPavel Emelyanov case TCP_QUEUE_SEQ: 3247ee995283SPavel Emelyanov if (tp->repair_queue == TCP_SEND_QUEUE) 3248ee995283SPavel Emelyanov val = tp->write_seq; 3249ee995283SPavel Emelyanov else if (tp->repair_queue == TCP_RECV_QUEUE) 3250ee995283SPavel Emelyanov val = tp->rcv_nxt; 3251ee995283SPavel Emelyanov else 3252ee995283SPavel Emelyanov return -EINVAL; 3253ee995283SPavel Emelyanov break; 3254ee995283SPavel Emelyanov 3255dca43c75SJerry Chu case TCP_USER_TIMEOUT: 3256dca43c75SJerry Chu val = jiffies_to_msecs(icsk->icsk_user_timeout); 3257dca43c75SJerry Chu break; 32581536e285SKenjiro Nakayama 32591536e285SKenjiro Nakayama case TCP_FASTOPEN: 32600536fcc0SEric Dumazet val = icsk->icsk_accept_queue.fastopenq.max_qlen; 32611536e285SKenjiro Nakayama break; 32621536e285SKenjiro Nakayama 326319f6d3f3SWei Wang case TCP_FASTOPEN_CONNECT: 326419f6d3f3SWei Wang val = tp->fastopen_connect; 326519f6d3f3SWei Wang break; 326619f6d3f3SWei Wang 3267*71c02379SChristoph Paasch case TCP_FASTOPEN_NO_COOKIE: 3268*71c02379SChristoph Paasch val = tp->fastopen_no_cookie; 3269*71c02379SChristoph Paasch break; 3270*71c02379SChristoph Paasch 327193be6ce0SAndrey Vagin case TCP_TIMESTAMP: 32729a568de4SEric Dumazet val = tcp_time_stamp_raw() + tp->tsoffset; 327393be6ce0SAndrey Vagin break; 3274c9bee3b7SEric Dumazet case TCP_NOTSENT_LOWAT: 3275c9bee3b7SEric Dumazet val = tp->notsent_lowat; 3276c9bee3b7SEric Dumazet break; 3277cd8ae852SEric Dumazet case TCP_SAVE_SYN: 3278cd8ae852SEric Dumazet val = tp->save_syn; 3279cd8ae852SEric Dumazet break; 3280cd8ae852SEric Dumazet case TCP_SAVED_SYN: { 3281cd8ae852SEric Dumazet if (get_user(len, optlen)) 3282cd8ae852SEric Dumazet return -EFAULT; 3283cd8ae852SEric Dumazet 3284cd8ae852SEric Dumazet lock_sock(sk); 3285cd8ae852SEric Dumazet if (tp->saved_syn) { 3286aea0929eSEric B Munson if (len < tp->saved_syn[0]) { 3287aea0929eSEric B Munson if (put_user(tp->saved_syn[0], optlen)) { 3288aea0929eSEric B Munson release_sock(sk); 3289aea0929eSEric B Munson return -EFAULT; 3290aea0929eSEric B Munson } 3291aea0929eSEric B Munson release_sock(sk); 3292aea0929eSEric B Munson return -EINVAL; 3293aea0929eSEric B Munson } 3294aea0929eSEric B Munson len = tp->saved_syn[0]; 3295cd8ae852SEric Dumazet if (put_user(len, optlen)) { 3296cd8ae852SEric Dumazet release_sock(sk); 3297cd8ae852SEric Dumazet return -EFAULT; 3298cd8ae852SEric Dumazet } 3299cd8ae852SEric Dumazet if (copy_to_user(optval, tp->saved_syn + 1, len)) { 3300cd8ae852SEric Dumazet release_sock(sk); 3301cd8ae852SEric Dumazet return -EFAULT; 3302cd8ae852SEric Dumazet } 3303cd8ae852SEric Dumazet tcp_saved_syn_free(tp); 3304cd8ae852SEric Dumazet release_sock(sk); 3305cd8ae852SEric Dumazet } else { 3306cd8ae852SEric Dumazet release_sock(sk); 3307cd8ae852SEric Dumazet len = 0; 3308cd8ae852SEric Dumazet if (put_user(len, optlen)) 3309cd8ae852SEric Dumazet return -EFAULT; 3310cd8ae852SEric Dumazet } 3311cd8ae852SEric Dumazet return 0; 3312cd8ae852SEric Dumazet } 33131da177e4SLinus Torvalds default: 33141da177e4SLinus Torvalds return -ENOPROTOOPT; 33153ff50b79SStephen Hemminger } 33161da177e4SLinus Torvalds 33171da177e4SLinus Torvalds if (put_user(len, optlen)) 33181da177e4SLinus Torvalds return -EFAULT; 33191da177e4SLinus Torvalds if (copy_to_user(optval, &val, len)) 33201da177e4SLinus Torvalds return -EFAULT; 33211da177e4SLinus Torvalds return 0; 33221da177e4SLinus Torvalds } 33231da177e4SLinus Torvalds 33243fdadf7dSDmitry Mishin int tcp_getsockopt(struct sock *sk, int level, int optname, char __user *optval, 33253fdadf7dSDmitry Mishin int __user *optlen) 33263fdadf7dSDmitry Mishin { 33273fdadf7dSDmitry Mishin struct inet_connection_sock *icsk = inet_csk(sk); 33283fdadf7dSDmitry Mishin 33293fdadf7dSDmitry Mishin if (level != SOL_TCP) 33303fdadf7dSDmitry Mishin return icsk->icsk_af_ops->getsockopt(sk, level, optname, 33313fdadf7dSDmitry Mishin optval, optlen); 33323fdadf7dSDmitry Mishin return do_tcp_getsockopt(sk, level, optname, optval, optlen); 33333fdadf7dSDmitry Mishin } 33344bc2f18bSEric Dumazet EXPORT_SYMBOL(tcp_getsockopt); 33353fdadf7dSDmitry Mishin 33363fdadf7dSDmitry Mishin #ifdef CONFIG_COMPAT 3337543d9cfeSArnaldo Carvalho de Melo int compat_tcp_getsockopt(struct sock *sk, int level, int optname, 3338543d9cfeSArnaldo Carvalho de Melo char __user *optval, int __user *optlen) 33393fdadf7dSDmitry Mishin { 3340dec73ff0SArnaldo Carvalho de Melo if (level != SOL_TCP) 3341dec73ff0SArnaldo Carvalho de Melo return inet_csk_compat_getsockopt(sk, level, optname, 3342dec73ff0SArnaldo Carvalho de Melo optval, optlen); 33433fdadf7dSDmitry Mishin return do_tcp_getsockopt(sk, level, optname, optval, optlen); 33443fdadf7dSDmitry Mishin } 3345543d9cfeSArnaldo Carvalho de Melo EXPORT_SYMBOL(compat_tcp_getsockopt); 33463fdadf7dSDmitry Mishin #endif 33471da177e4SLinus Torvalds 3348cfb6eeb4SYOSHIFUJI Hideaki #ifdef CONFIG_TCP_MD5SIG 3349349ce993SEric Dumazet static DEFINE_PER_CPU(struct tcp_md5sig_pool, tcp_md5sig_pool); 335071cea17eSEric Dumazet static DEFINE_MUTEX(tcp_md5sig_mutex); 3351349ce993SEric Dumazet static bool tcp_md5sig_pool_populated = false; 3352cfb6eeb4SYOSHIFUJI Hideaki 335371cea17eSEric Dumazet static void __tcp_alloc_md5sig_pool(void) 3354cfb6eeb4SYOSHIFUJI Hideaki { 3355cf80e0e4SHerbert Xu struct crypto_ahash *hash; 3356cfb6eeb4SYOSHIFUJI Hideaki int cpu; 3357cfb6eeb4SYOSHIFUJI Hideaki 3358cf80e0e4SHerbert Xu hash = crypto_alloc_ahash("md5", 0, CRYPTO_ALG_ASYNC); 33591eea84b7SInsu Yun if (IS_ERR(hash)) 3360349ce993SEric Dumazet return; 3361cf80e0e4SHerbert Xu 3362cf80e0e4SHerbert Xu for_each_possible_cpu(cpu) { 336319689e38SEric Dumazet void *scratch = per_cpu(tcp_md5sig_pool, cpu).scratch; 3364cf80e0e4SHerbert Xu struct ahash_request *req; 3365cf80e0e4SHerbert Xu 336619689e38SEric Dumazet if (!scratch) { 336719689e38SEric Dumazet scratch = kmalloc_node(sizeof(union tcp_md5sum_block) + 336819689e38SEric Dumazet sizeof(struct tcphdr), 336919689e38SEric Dumazet GFP_KERNEL, 337019689e38SEric Dumazet cpu_to_node(cpu)); 337119689e38SEric Dumazet if (!scratch) 337219689e38SEric Dumazet return; 337319689e38SEric Dumazet per_cpu(tcp_md5sig_pool, cpu).scratch = scratch; 337419689e38SEric Dumazet } 3375cf80e0e4SHerbert Xu if (per_cpu(tcp_md5sig_pool, cpu).md5_req) 3376cf80e0e4SHerbert Xu continue; 3377cf80e0e4SHerbert Xu 3378cf80e0e4SHerbert Xu req = ahash_request_alloc(hash, GFP_KERNEL); 3379cf80e0e4SHerbert Xu if (!req) 3380cf80e0e4SHerbert Xu return; 3381cf80e0e4SHerbert Xu 3382cf80e0e4SHerbert Xu ahash_request_set_callback(req, 0, NULL, NULL); 3383cf80e0e4SHerbert Xu 3384cf80e0e4SHerbert Xu per_cpu(tcp_md5sig_pool, cpu).md5_req = req; 3385349ce993SEric Dumazet } 3386349ce993SEric Dumazet /* before setting tcp_md5sig_pool_populated, we must commit all writes 3387349ce993SEric Dumazet * to memory. See smp_rmb() in tcp_get_md5sig_pool() 338871cea17eSEric Dumazet */ 338971cea17eSEric Dumazet smp_wmb(); 3390349ce993SEric Dumazet tcp_md5sig_pool_populated = true; 3391cfb6eeb4SYOSHIFUJI Hideaki } 3392cfb6eeb4SYOSHIFUJI Hideaki 339371cea17eSEric Dumazet bool tcp_alloc_md5sig_pool(void) 3394cfb6eeb4SYOSHIFUJI Hideaki { 3395349ce993SEric Dumazet if (unlikely(!tcp_md5sig_pool_populated)) { 339671cea17eSEric Dumazet mutex_lock(&tcp_md5sig_mutex); 3397cfb6eeb4SYOSHIFUJI Hideaki 3398349ce993SEric Dumazet if (!tcp_md5sig_pool_populated) 339971cea17eSEric Dumazet __tcp_alloc_md5sig_pool(); 3400cfb6eeb4SYOSHIFUJI Hideaki 340171cea17eSEric Dumazet mutex_unlock(&tcp_md5sig_mutex); 3402cfb6eeb4SYOSHIFUJI Hideaki } 3403349ce993SEric Dumazet return tcp_md5sig_pool_populated; 3404cfb6eeb4SYOSHIFUJI Hideaki } 3405cfb6eeb4SYOSHIFUJI Hideaki EXPORT_SYMBOL(tcp_alloc_md5sig_pool); 3406cfb6eeb4SYOSHIFUJI Hideaki 340735790c04SEric Dumazet 340835790c04SEric Dumazet /** 340935790c04SEric Dumazet * tcp_get_md5sig_pool - get md5sig_pool for this user 341035790c04SEric Dumazet * 341135790c04SEric Dumazet * We use percpu structure, so if we succeed, we exit with preemption 341235790c04SEric Dumazet * and BH disabled, to make sure another thread or softirq handling 341335790c04SEric Dumazet * wont try to get same context. 341435790c04SEric Dumazet */ 341535790c04SEric Dumazet struct tcp_md5sig_pool *tcp_get_md5sig_pool(void) 3416cfb6eeb4SYOSHIFUJI Hideaki { 341735790c04SEric Dumazet local_bh_disable(); 341835790c04SEric Dumazet 3419349ce993SEric Dumazet if (tcp_md5sig_pool_populated) { 3420349ce993SEric Dumazet /* coupled with smp_wmb() in __tcp_alloc_md5sig_pool() */ 3421349ce993SEric Dumazet smp_rmb(); 3422349ce993SEric Dumazet return this_cpu_ptr(&tcp_md5sig_pool); 3423349ce993SEric Dumazet } 342435790c04SEric Dumazet local_bh_enable(); 342535790c04SEric Dumazet return NULL; 3426cfb6eeb4SYOSHIFUJI Hideaki } 342735790c04SEric Dumazet EXPORT_SYMBOL(tcp_get_md5sig_pool); 3428cfb6eeb4SYOSHIFUJI Hideaki 342949a72dfbSAdam Langley int tcp_md5_hash_skb_data(struct tcp_md5sig_pool *hp, 3430cf533ea5SEric Dumazet const struct sk_buff *skb, unsigned int header_len) 343149a72dfbSAdam Langley { 343249a72dfbSAdam Langley struct scatterlist sg; 343349a72dfbSAdam Langley const struct tcphdr *tp = tcp_hdr(skb); 3434cf80e0e4SHerbert Xu struct ahash_request *req = hp->md5_req; 343595c96174SEric Dumazet unsigned int i; 343695c96174SEric Dumazet const unsigned int head_data_len = skb_headlen(skb) > header_len ? 343749a72dfbSAdam Langley skb_headlen(skb) - header_len : 0; 343849a72dfbSAdam Langley const struct skb_shared_info *shi = skb_shinfo(skb); 3439d7fd1b57SEric Dumazet struct sk_buff *frag_iter; 344049a72dfbSAdam Langley 344149a72dfbSAdam Langley sg_init_table(&sg, 1); 344249a72dfbSAdam Langley 344349a72dfbSAdam Langley sg_set_buf(&sg, ((u8 *) tp) + header_len, head_data_len); 3444cf80e0e4SHerbert Xu ahash_request_set_crypt(req, &sg, NULL, head_data_len); 3445cf80e0e4SHerbert Xu if (crypto_ahash_update(req)) 344649a72dfbSAdam Langley return 1; 344749a72dfbSAdam Langley 344849a72dfbSAdam Langley for (i = 0; i < shi->nr_frags; ++i) { 344949a72dfbSAdam Langley const struct skb_frag_struct *f = &shi->frags[i]; 345054d27fcbSEric Dumazet unsigned int offset = f->page_offset; 345154d27fcbSEric Dumazet struct page *page = skb_frag_page(f) + (offset >> PAGE_SHIFT); 345254d27fcbSEric Dumazet 345354d27fcbSEric Dumazet sg_set_page(&sg, page, skb_frag_size(f), 345454d27fcbSEric Dumazet offset_in_page(offset)); 3455cf80e0e4SHerbert Xu ahash_request_set_crypt(req, &sg, NULL, skb_frag_size(f)); 3456cf80e0e4SHerbert Xu if (crypto_ahash_update(req)) 345749a72dfbSAdam Langley return 1; 345849a72dfbSAdam Langley } 345949a72dfbSAdam Langley 3460d7fd1b57SEric Dumazet skb_walk_frags(skb, frag_iter) 3461d7fd1b57SEric Dumazet if (tcp_md5_hash_skb_data(hp, frag_iter, 0)) 3462d7fd1b57SEric Dumazet return 1; 3463d7fd1b57SEric Dumazet 346449a72dfbSAdam Langley return 0; 346549a72dfbSAdam Langley } 346649a72dfbSAdam Langley EXPORT_SYMBOL(tcp_md5_hash_skb_data); 346749a72dfbSAdam Langley 3468cf533ea5SEric Dumazet int tcp_md5_hash_key(struct tcp_md5sig_pool *hp, const struct tcp_md5sig_key *key) 346949a72dfbSAdam Langley { 347049a72dfbSAdam Langley struct scatterlist sg; 347149a72dfbSAdam Langley 347249a72dfbSAdam Langley sg_init_one(&sg, key->key, key->keylen); 3473cf80e0e4SHerbert Xu ahash_request_set_crypt(hp->md5_req, &sg, NULL, key->keylen); 3474cf80e0e4SHerbert Xu return crypto_ahash_update(hp->md5_req); 347549a72dfbSAdam Langley } 347649a72dfbSAdam Langley EXPORT_SYMBOL(tcp_md5_hash_key); 347749a72dfbSAdam Langley 3478cfb6eeb4SYOSHIFUJI Hideaki #endif 3479cfb6eeb4SYOSHIFUJI Hideaki 34804ac02babSAndi Kleen void tcp_done(struct sock *sk) 34814ac02babSAndi Kleen { 34828336886fSJerry Chu struct request_sock *req = tcp_sk(sk)->fastopen_rsk; 34838336886fSJerry Chu 34844ac02babSAndi Kleen if (sk->sk_state == TCP_SYN_SENT || sk->sk_state == TCP_SYN_RECV) 3485c10d9310SEric Dumazet TCP_INC_STATS(sock_net(sk), TCP_MIB_ATTEMPTFAILS); 34864ac02babSAndi Kleen 34874ac02babSAndi Kleen tcp_set_state(sk, TCP_CLOSE); 34884ac02babSAndi Kleen tcp_clear_xmit_timers(sk); 348900db4124SIan Morris if (req) 34908336886fSJerry Chu reqsk_fastopen_remove(sk, req, false); 34914ac02babSAndi Kleen 34924ac02babSAndi Kleen sk->sk_shutdown = SHUTDOWN_MASK; 34934ac02babSAndi Kleen 34944ac02babSAndi Kleen if (!sock_flag(sk, SOCK_DEAD)) 34954ac02babSAndi Kleen sk->sk_state_change(sk); 34964ac02babSAndi Kleen else 34974ac02babSAndi Kleen inet_csk_destroy_sock(sk); 34984ac02babSAndi Kleen } 34994ac02babSAndi Kleen EXPORT_SYMBOL_GPL(tcp_done); 35004ac02babSAndi Kleen 3501c1e64e29SLorenzo Colitti int tcp_abort(struct sock *sk, int err) 3502c1e64e29SLorenzo Colitti { 3503c1e64e29SLorenzo Colitti if (!sk_fullsock(sk)) { 350407f6f4a3SEric Dumazet if (sk->sk_state == TCP_NEW_SYN_RECV) { 350507f6f4a3SEric Dumazet struct request_sock *req = inet_reqsk(sk); 350607f6f4a3SEric Dumazet 350707f6f4a3SEric Dumazet local_bh_disable(); 350807f6f4a3SEric Dumazet inet_csk_reqsk_queue_drop_and_put(req->rsk_listener, 350907f6f4a3SEric Dumazet req); 351007f6f4a3SEric Dumazet local_bh_enable(); 351107f6f4a3SEric Dumazet return 0; 351207f6f4a3SEric Dumazet } 3513c1e64e29SLorenzo Colitti return -EOPNOTSUPP; 3514c1e64e29SLorenzo Colitti } 3515c1e64e29SLorenzo Colitti 3516c1e64e29SLorenzo Colitti /* Don't race with userspace socket closes such as tcp_close. */ 3517c1e64e29SLorenzo Colitti lock_sock(sk); 3518c1e64e29SLorenzo Colitti 35192010b93eSLorenzo Colitti if (sk->sk_state == TCP_LISTEN) { 35202010b93eSLorenzo Colitti tcp_set_state(sk, TCP_CLOSE); 35212010b93eSLorenzo Colitti inet_csk_listen_stop(sk); 35222010b93eSLorenzo Colitti } 35232010b93eSLorenzo Colitti 3524c1e64e29SLorenzo Colitti /* Don't race with BH socket closes such as inet_csk_listen_stop. */ 3525c1e64e29SLorenzo Colitti local_bh_disable(); 3526c1e64e29SLorenzo Colitti bh_lock_sock(sk); 3527c1e64e29SLorenzo Colitti 3528c1e64e29SLorenzo Colitti if (!sock_flag(sk, SOCK_DEAD)) { 3529c1e64e29SLorenzo Colitti sk->sk_err = err; 3530c1e64e29SLorenzo Colitti /* This barrier is coupled with smp_rmb() in tcp_poll() */ 3531c1e64e29SLorenzo Colitti smp_wmb(); 3532c1e64e29SLorenzo Colitti sk->sk_error_report(sk); 3533c1e64e29SLorenzo Colitti if (tcp_need_reset(sk->sk_state)) 3534c1e64e29SLorenzo Colitti tcp_send_active_reset(sk, GFP_ATOMIC); 3535c1e64e29SLorenzo Colitti tcp_done(sk); 3536c1e64e29SLorenzo Colitti } 3537c1e64e29SLorenzo Colitti 3538c1e64e29SLorenzo Colitti bh_unlock_sock(sk); 3539c1e64e29SLorenzo Colitti local_bh_enable(); 3540c1e64e29SLorenzo Colitti release_sock(sk); 3541c1e64e29SLorenzo Colitti return 0; 3542c1e64e29SLorenzo Colitti } 3543c1e64e29SLorenzo Colitti EXPORT_SYMBOL_GPL(tcp_abort); 3544c1e64e29SLorenzo Colitti 35455f8ef48dSStephen Hemminger extern struct tcp_congestion_ops tcp_reno; 35461da177e4SLinus Torvalds 35471da177e4SLinus Torvalds static __initdata unsigned long thash_entries; 35481da177e4SLinus Torvalds static int __init set_thash_entries(char *str) 35491da177e4SLinus Torvalds { 3550413c27d8SEldad Zack ssize_t ret; 3551413c27d8SEldad Zack 35521da177e4SLinus Torvalds if (!str) 35531da177e4SLinus Torvalds return 0; 3554413c27d8SEldad Zack 3555413c27d8SEldad Zack ret = kstrtoul(str, 0, &thash_entries); 3556413c27d8SEldad Zack if (ret) 3557413c27d8SEldad Zack return 0; 3558413c27d8SEldad Zack 35591da177e4SLinus Torvalds return 1; 35601da177e4SLinus Torvalds } 35611da177e4SLinus Torvalds __setup("thash_entries=", set_thash_entries); 35621da177e4SLinus Torvalds 356347d7a88cSFabian Frederick static void __init tcp_init_mem(void) 35644acb4190SGlauber Costa { 3565b66e91ccSEric Dumazet unsigned long limit = nr_free_buffer_pages() / 16; 3566b66e91ccSEric Dumazet 35674acb4190SGlauber Costa limit = max(limit, 128UL); 3568b66e91ccSEric Dumazet sysctl_tcp_mem[0] = limit / 4 * 3; /* 4.68 % */ 3569b66e91ccSEric Dumazet sysctl_tcp_mem[1] = limit; /* 6.25 % */ 3570b66e91ccSEric Dumazet sysctl_tcp_mem[2] = sysctl_tcp_mem[0] * 2; /* 9.37 % */ 35714acb4190SGlauber Costa } 35724acb4190SGlauber Costa 35731da177e4SLinus Torvalds void __init tcp_init(void) 35741da177e4SLinus Torvalds { 3575b49960a0SEric Dumazet int max_rshare, max_wshare, cnt; 3576b2d3ea4aSEric Dumazet unsigned long limit; 3577074b8517SDimitri Sivanich unsigned int i; 35781da177e4SLinus Torvalds 3579b2d3ea4aSEric Dumazet BUILD_BUG_ON(sizeof(struct tcp_skb_cb) > 3580b2d3ea4aSEric Dumazet FIELD_SIZEOF(struct sk_buff, cb)); 35811da177e4SLinus Torvalds 3582908c7f19STejun Heo percpu_counter_init(&tcp_sockets_allocated, 0, GFP_KERNEL); 3583908c7f19STejun Heo percpu_counter_init(&tcp_orphan_count, 0, GFP_KERNEL); 35841946e672SHaishuang Yan inet_hashinfo_init(&tcp_hashinfo); 35856e04e021SArnaldo Carvalho de Melo tcp_hashinfo.bind_bucket_cachep = 35866e04e021SArnaldo Carvalho de Melo kmem_cache_create("tcp_bind_bucket", 35876e04e021SArnaldo Carvalho de Melo sizeof(struct inet_bind_bucket), 0, 358820c2df83SPaul Mundt SLAB_HWCACHE_ALIGN|SLAB_PANIC, NULL); 35891da177e4SLinus Torvalds 35901da177e4SLinus Torvalds /* Size and allocate the main established and bind bucket 35911da177e4SLinus Torvalds * hash tables. 35921da177e4SLinus Torvalds * 35931da177e4SLinus Torvalds * The methodology is similar to that of the buffer cache. 35941da177e4SLinus Torvalds */ 35956e04e021SArnaldo Carvalho de Melo tcp_hashinfo.ehash = 35961da177e4SLinus Torvalds alloc_large_system_hash("TCP established", 35970f7ff927SArnaldo Carvalho de Melo sizeof(struct inet_ehash_bucket), 35981da177e4SLinus Torvalds thash_entries, 3599fd90b29dSEric Dumazet 17, /* one slot per 128 KB of memory */ 36009e950efaSJohn Heffner 0, 36011da177e4SLinus Torvalds NULL, 3602f373b53bSEric Dumazet &tcp_hashinfo.ehash_mask, 360331fe62b9STim Bird 0, 36040ccfe618SJean Delvare thash_entries ? 0 : 512 * 1024); 360505dbc7b5SEric Dumazet for (i = 0; i <= tcp_hashinfo.ehash_mask; i++) 36063ab5aee7SEric Dumazet INIT_HLIST_NULLS_HEAD(&tcp_hashinfo.ehash[i].chain, i); 360705dbc7b5SEric Dumazet 3608230140cfSEric Dumazet if (inet_ehash_locks_alloc(&tcp_hashinfo)) 3609230140cfSEric Dumazet panic("TCP: failed to alloc ehash_locks"); 36106e04e021SArnaldo Carvalho de Melo tcp_hashinfo.bhash = 36111da177e4SLinus Torvalds alloc_large_system_hash("TCP bind", 36120f7ff927SArnaldo Carvalho de Melo sizeof(struct inet_bind_hashbucket), 3613f373b53bSEric Dumazet tcp_hashinfo.ehash_mask + 1, 3614fd90b29dSEric Dumazet 17, /* one slot per 128 KB of memory */ 36159e950efaSJohn Heffner 0, 36166e04e021SArnaldo Carvalho de Melo &tcp_hashinfo.bhash_size, 36171da177e4SLinus Torvalds NULL, 361831fe62b9STim Bird 0, 36191da177e4SLinus Torvalds 64 * 1024); 3620074b8517SDimitri Sivanich tcp_hashinfo.bhash_size = 1U << tcp_hashinfo.bhash_size; 36216e04e021SArnaldo Carvalho de Melo for (i = 0; i < tcp_hashinfo.bhash_size; i++) { 36226e04e021SArnaldo Carvalho de Melo spin_lock_init(&tcp_hashinfo.bhash[i].lock); 36236e04e021SArnaldo Carvalho de Melo INIT_HLIST_HEAD(&tcp_hashinfo.bhash[i].chain); 36241da177e4SLinus Torvalds } 36251da177e4SLinus Torvalds 3626c5ed63d6SEric Dumazet 3627c5ed63d6SEric Dumazet cnt = tcp_hashinfo.ehash_mask + 1; 3628c5ed63d6SEric Dumazet sysctl_tcp_max_orphans = cnt / 2; 36291da177e4SLinus Torvalds 3630a4fe34bfSEric W. Biederman tcp_init_mem(); 3631c43b874dSJason Wang /* Set per-socket limits to no more than 1/128 the pressure threshold */ 36325fb84b14SEric Dumazet limit = nr_free_buffer_pages() << (PAGE_SHIFT - 7); 3633b49960a0SEric Dumazet max_wshare = min(4UL*1024*1024, limit); 3634b49960a0SEric Dumazet max_rshare = min(6UL*1024*1024, limit); 36357b4f4b5eSJohn Heffner 36363ab224beSHideo Aoki sysctl_tcp_wmem[0] = SK_MEM_QUANTUM; 36377b4f4b5eSJohn Heffner sysctl_tcp_wmem[1] = 16*1024; 3638b49960a0SEric Dumazet sysctl_tcp_wmem[2] = max(64*1024, max_wshare); 36397b4f4b5eSJohn Heffner 36403ab224beSHideo Aoki sysctl_tcp_rmem[0] = SK_MEM_QUANTUM; 36417b4f4b5eSJohn Heffner sysctl_tcp_rmem[1] = 87380; 3642b49960a0SEric Dumazet sysctl_tcp_rmem[2] = max(87380, max_rshare); 36431da177e4SLinus Torvalds 3644afd46503SJoe Perches pr_info("Hash tables configured (established %u bind %u)\n", 3645f373b53bSEric Dumazet tcp_hashinfo.ehash_mask + 1, tcp_hashinfo.bhash_size); 3646317a76f9SStephen Hemminger 36471946e672SHaishuang Yan tcp_v4_init(); 364851c5d0c4SDavid S. Miller tcp_metrics_init(); 364955d8694fSFlorian Westphal BUG_ON(tcp_register_congestion_control(&tcp_reno) != 0); 365046d3ceabSEric Dumazet tcp_tasklet_init(); 36511da177e4SLinus Torvalds } 3652