11da177e4SLinus Torvalds /* 21da177e4SLinus Torvalds * INET An implementation of the TCP/IP protocol suite for the LINUX 31da177e4SLinus Torvalds * operating system. INET is implemented using the BSD Socket 41da177e4SLinus Torvalds * interface as the means of communication with the user level. 51da177e4SLinus Torvalds * 61da177e4SLinus Torvalds * Implementation of the Transmission Control Protocol(TCP). 71da177e4SLinus Torvalds * 802c30a84SJesper Juhl * Authors: Ross Biro 91da177e4SLinus Torvalds * Fred N. van Kempen, <waltje@uWalt.NL.Mugnet.ORG> 101da177e4SLinus Torvalds * Mark Evans, <evansmp@uhura.aston.ac.uk> 111da177e4SLinus Torvalds * Corey Minyard <wf-rch!minyard@relay.EU.net> 121da177e4SLinus Torvalds * Florian La Roche, <flla@stud.uni-sb.de> 131da177e4SLinus Torvalds * Charles Hedrick, <hedrick@klinzhai.rutgers.edu> 141da177e4SLinus Torvalds * Linus Torvalds, <torvalds@cs.helsinki.fi> 151da177e4SLinus Torvalds * Alan Cox, <gw4pts@gw4pts.ampr.org> 161da177e4SLinus Torvalds * Matthew Dillon, <dillon@apollo.west.oic.com> 171da177e4SLinus Torvalds * Arnt Gulbrandsen, <agulbra@nvg.unit.no> 181da177e4SLinus Torvalds * Jorge Cwik, <jorge@laser.satlink.net> 191da177e4SLinus Torvalds * 201da177e4SLinus Torvalds * Fixes: 211da177e4SLinus Torvalds * Alan Cox : Numerous verify_area() calls 221da177e4SLinus Torvalds * Alan Cox : Set the ACK bit on a reset 231da177e4SLinus Torvalds * Alan Cox : Stopped it crashing if it closed while 241da177e4SLinus Torvalds * sk->inuse=1 and was trying to connect 251da177e4SLinus Torvalds * (tcp_err()). 261da177e4SLinus Torvalds * Alan Cox : All icmp error handling was broken 271da177e4SLinus Torvalds * pointers passed where wrong and the 281da177e4SLinus Torvalds * socket was looked up backwards. Nobody 291da177e4SLinus Torvalds * tested any icmp error code obviously. 301da177e4SLinus Torvalds * Alan Cox : tcp_err() now handled properly. It 311da177e4SLinus Torvalds * wakes people on errors. poll 321da177e4SLinus Torvalds * behaves and the icmp error race 331da177e4SLinus Torvalds * has gone by moving it into sock.c 341da177e4SLinus Torvalds * Alan Cox : tcp_send_reset() fixed to work for 351da177e4SLinus Torvalds * everything not just packets for 361da177e4SLinus Torvalds * unknown sockets. 371da177e4SLinus Torvalds * Alan Cox : tcp option processing. 381da177e4SLinus Torvalds * Alan Cox : Reset tweaked (still not 100%) [Had 391da177e4SLinus Torvalds * syn rule wrong] 401da177e4SLinus Torvalds * Herp Rosmanith : More reset fixes 411da177e4SLinus Torvalds * Alan Cox : No longer acks invalid rst frames. 421da177e4SLinus Torvalds * Acking any kind of RST is right out. 431da177e4SLinus Torvalds * Alan Cox : Sets an ignore me flag on an rst 441da177e4SLinus Torvalds * receive otherwise odd bits of prattle 451da177e4SLinus Torvalds * escape still 461da177e4SLinus Torvalds * Alan Cox : Fixed another acking RST frame bug. 471da177e4SLinus Torvalds * Should stop LAN workplace lockups. 481da177e4SLinus Torvalds * Alan Cox : Some tidyups using the new skb list 491da177e4SLinus Torvalds * facilities 501da177e4SLinus Torvalds * Alan Cox : sk->keepopen now seems to work 511da177e4SLinus Torvalds * Alan Cox : Pulls options out correctly on accepts 521da177e4SLinus Torvalds * Alan Cox : Fixed assorted sk->rqueue->next errors 531da177e4SLinus Torvalds * Alan Cox : PSH doesn't end a TCP read. Switched a 541da177e4SLinus Torvalds * bit to skb ops. 551da177e4SLinus Torvalds * Alan Cox : Tidied tcp_data to avoid a potential 561da177e4SLinus Torvalds * nasty. 571da177e4SLinus Torvalds * Alan Cox : Added some better commenting, as the 581da177e4SLinus Torvalds * tcp is hard to follow 591da177e4SLinus Torvalds * Alan Cox : Removed incorrect check for 20 * psh 601da177e4SLinus Torvalds * Michael O'Reilly : ack < copied bug fix. 611da177e4SLinus Torvalds * Johannes Stille : Misc tcp fixes (not all in yet). 621da177e4SLinus Torvalds * Alan Cox : FIN with no memory -> CRASH 631da177e4SLinus Torvalds * Alan Cox : Added socket option proto entries. 641da177e4SLinus Torvalds * Also added awareness of them to accept. 651da177e4SLinus Torvalds * Alan Cox : Added TCP options (SOL_TCP) 661da177e4SLinus Torvalds * Alan Cox : Switched wakeup calls to callbacks, 671da177e4SLinus Torvalds * so the kernel can layer network 681da177e4SLinus Torvalds * sockets. 691da177e4SLinus Torvalds * Alan Cox : Use ip_tos/ip_ttl settings. 701da177e4SLinus Torvalds * Alan Cox : Handle FIN (more) properly (we hope). 711da177e4SLinus Torvalds * Alan Cox : RST frames sent on unsynchronised 721da177e4SLinus Torvalds * state ack error. 731da177e4SLinus Torvalds * Alan Cox : Put in missing check for SYN bit. 741da177e4SLinus Torvalds * Alan Cox : Added tcp_select_window() aka NET2E 751da177e4SLinus Torvalds * window non shrink trick. 761da177e4SLinus Torvalds * Alan Cox : Added a couple of small NET2E timer 771da177e4SLinus Torvalds * fixes 781da177e4SLinus Torvalds * Charles Hedrick : TCP fixes 791da177e4SLinus Torvalds * Toomas Tamm : TCP window fixes 801da177e4SLinus Torvalds * Alan Cox : Small URG fix to rlogin ^C ack fight 811da177e4SLinus Torvalds * Charles Hedrick : Rewrote most of it to actually work 821da177e4SLinus Torvalds * Linus : Rewrote tcp_read() and URG handling 831da177e4SLinus Torvalds * completely 841da177e4SLinus Torvalds * Gerhard Koerting: Fixed some missing timer handling 851da177e4SLinus Torvalds * Matthew Dillon : Reworked TCP machine states as per RFC 861da177e4SLinus Torvalds * Gerhard Koerting: PC/TCP workarounds 871da177e4SLinus Torvalds * Adam Caldwell : Assorted timer/timing errors 881da177e4SLinus Torvalds * Matthew Dillon : Fixed another RST bug 891da177e4SLinus Torvalds * Alan Cox : Move to kernel side addressing changes. 901da177e4SLinus Torvalds * Alan Cox : Beginning work on TCP fastpathing 911da177e4SLinus Torvalds * (not yet usable) 921da177e4SLinus Torvalds * Arnt Gulbrandsen: Turbocharged tcp_check() routine. 931da177e4SLinus Torvalds * Alan Cox : TCP fast path debugging 941da177e4SLinus Torvalds * Alan Cox : Window clamping 951da177e4SLinus Torvalds * Michael Riepe : Bug in tcp_check() 961da177e4SLinus Torvalds * Matt Dillon : More TCP improvements and RST bug fixes 971da177e4SLinus Torvalds * Matt Dillon : Yet more small nasties remove from the 981da177e4SLinus Torvalds * TCP code (Be very nice to this man if 991da177e4SLinus Torvalds * tcp finally works 100%) 8) 1001da177e4SLinus Torvalds * Alan Cox : BSD accept semantics. 1011da177e4SLinus Torvalds * Alan Cox : Reset on closedown bug. 1021da177e4SLinus Torvalds * Peter De Schrijver : ENOTCONN check missing in tcp_sendto(). 1031da177e4SLinus Torvalds * Michael Pall : Handle poll() after URG properly in 1041da177e4SLinus Torvalds * all cases. 1051da177e4SLinus Torvalds * Michael Pall : Undo the last fix in tcp_read_urg() 1061da177e4SLinus Torvalds * (multi URG PUSH broke rlogin). 1071da177e4SLinus Torvalds * Michael Pall : Fix the multi URG PUSH problem in 1081da177e4SLinus Torvalds * tcp_readable(), poll() after URG 1091da177e4SLinus Torvalds * works now. 1101da177e4SLinus Torvalds * Michael Pall : recv(...,MSG_OOB) never blocks in the 1111da177e4SLinus Torvalds * BSD api. 1121da177e4SLinus Torvalds * Alan Cox : Changed the semantics of sk->socket to 1131da177e4SLinus Torvalds * fix a race and a signal problem with 1141da177e4SLinus Torvalds * accept() and async I/O. 1151da177e4SLinus Torvalds * Alan Cox : Relaxed the rules on tcp_sendto(). 1161da177e4SLinus Torvalds * Yury Shevchuk : Really fixed accept() blocking problem. 1171da177e4SLinus Torvalds * Craig I. Hagan : Allow for BSD compatible TIME_WAIT for 1181da177e4SLinus Torvalds * clients/servers which listen in on 1191da177e4SLinus Torvalds * fixed ports. 1201da177e4SLinus Torvalds * Alan Cox : Cleaned the above up and shrank it to 1211da177e4SLinus Torvalds * a sensible code size. 1221da177e4SLinus Torvalds * Alan Cox : Self connect lockup fix. 1231da177e4SLinus Torvalds * Alan Cox : No connect to multicast. 1241da177e4SLinus Torvalds * Ross Biro : Close unaccepted children on master 1251da177e4SLinus Torvalds * socket close. 1261da177e4SLinus Torvalds * Alan Cox : Reset tracing code. 1271da177e4SLinus Torvalds * Alan Cox : Spurious resets on shutdown. 1281da177e4SLinus Torvalds * Alan Cox : Giant 15 minute/60 second timer error 1291da177e4SLinus Torvalds * Alan Cox : Small whoops in polling before an 1301da177e4SLinus Torvalds * accept. 1311da177e4SLinus Torvalds * Alan Cox : Kept the state trace facility since 1321da177e4SLinus Torvalds * it's handy for debugging. 1331da177e4SLinus Torvalds * Alan Cox : More reset handler fixes. 1341da177e4SLinus Torvalds * Alan Cox : Started rewriting the code based on 1351da177e4SLinus Torvalds * the RFC's for other useful protocol 1361da177e4SLinus Torvalds * references see: Comer, KA9Q NOS, and 1371da177e4SLinus Torvalds * for a reference on the difference 1381da177e4SLinus Torvalds * between specifications and how BSD 1391da177e4SLinus Torvalds * works see the 4.4lite source. 1401da177e4SLinus Torvalds * A.N.Kuznetsov : Don't time wait on completion of tidy 1411da177e4SLinus Torvalds * close. 1421da177e4SLinus Torvalds * Linus Torvalds : Fin/Shutdown & copied_seq changes. 1431da177e4SLinus Torvalds * Linus Torvalds : Fixed BSD port reuse to work first syn 1441da177e4SLinus Torvalds * Alan Cox : Reimplemented timers as per the RFC 1451da177e4SLinus Torvalds * and using multiple timers for sanity. 1461da177e4SLinus Torvalds * Alan Cox : Small bug fixes, and a lot of new 1471da177e4SLinus Torvalds * comments. 1481da177e4SLinus Torvalds * Alan Cox : Fixed dual reader crash by locking 1491da177e4SLinus Torvalds * the buffers (much like datagram.c) 1501da177e4SLinus Torvalds * Alan Cox : Fixed stuck sockets in probe. A probe 1511da177e4SLinus Torvalds * now gets fed up of retrying without 1521da177e4SLinus Torvalds * (even a no space) answer. 1531da177e4SLinus Torvalds * Alan Cox : Extracted closing code better 1541da177e4SLinus Torvalds * Alan Cox : Fixed the closing state machine to 1551da177e4SLinus Torvalds * resemble the RFC. 1561da177e4SLinus Torvalds * Alan Cox : More 'per spec' fixes. 1571da177e4SLinus Torvalds * Jorge Cwik : Even faster checksumming. 1581da177e4SLinus Torvalds * Alan Cox : tcp_data() doesn't ack illegal PSH 1591da177e4SLinus Torvalds * only frames. At least one pc tcp stack 1601da177e4SLinus Torvalds * generates them. 1611da177e4SLinus Torvalds * Alan Cox : Cache last socket. 1621da177e4SLinus Torvalds * Alan Cox : Per route irtt. 1631da177e4SLinus Torvalds * Matt Day : poll()->select() match BSD precisely on error 1641da177e4SLinus Torvalds * Alan Cox : New buffers 1651da177e4SLinus Torvalds * Marc Tamsky : Various sk->prot->retransmits and 1661da177e4SLinus Torvalds * sk->retransmits misupdating fixed. 1671da177e4SLinus Torvalds * Fixed tcp_write_timeout: stuck close, 1681da177e4SLinus Torvalds * and TCP syn retries gets used now. 1691da177e4SLinus Torvalds * Mark Yarvis : In tcp_read_wakeup(), don't send an 1701da177e4SLinus Torvalds * ack if state is TCP_CLOSED. 1711da177e4SLinus Torvalds * Alan Cox : Look up device on a retransmit - routes may 1721da177e4SLinus Torvalds * change. Doesn't yet cope with MSS shrink right 1731da177e4SLinus Torvalds * but it's a start! 1741da177e4SLinus Torvalds * Marc Tamsky : Closing in closing fixes. 1751da177e4SLinus Torvalds * Mike Shaver : RFC1122 verifications. 1761da177e4SLinus Torvalds * Alan Cox : rcv_saddr errors. 1771da177e4SLinus Torvalds * Alan Cox : Block double connect(). 1781da177e4SLinus Torvalds * Alan Cox : Small hooks for enSKIP. 1791da177e4SLinus Torvalds * Alexey Kuznetsov: Path MTU discovery. 1801da177e4SLinus Torvalds * Alan Cox : Support soft errors. 1811da177e4SLinus Torvalds * Alan Cox : Fix MTU discovery pathological case 1821da177e4SLinus Torvalds * when the remote claims no mtu! 1831da177e4SLinus Torvalds * Marc Tamsky : TCP_CLOSE fix. 1841da177e4SLinus Torvalds * Colin (G3TNE) : Send a reset on syn ack replies in 1851da177e4SLinus Torvalds * window but wrong (fixes NT lpd problems) 1861da177e4SLinus Torvalds * Pedro Roque : Better TCP window handling, delayed ack. 1871da177e4SLinus Torvalds * Joerg Reuter : No modification of locked buffers in 1881da177e4SLinus Torvalds * tcp_do_retransmit() 1891da177e4SLinus Torvalds * Eric Schenk : Changed receiver side silly window 1901da177e4SLinus Torvalds * avoidance algorithm to BSD style 1911da177e4SLinus Torvalds * algorithm. This doubles throughput 1921da177e4SLinus Torvalds * against machines running Solaris, 1931da177e4SLinus Torvalds * and seems to result in general 1941da177e4SLinus Torvalds * improvement. 1951da177e4SLinus Torvalds * Stefan Magdalinski : adjusted tcp_readable() to fix FIONREAD 1961da177e4SLinus Torvalds * Willy Konynenberg : Transparent proxying support. 1971da177e4SLinus Torvalds * Mike McLagan : Routing by source 1981da177e4SLinus Torvalds * Keith Owens : Do proper merging with partial SKB's in 1991da177e4SLinus Torvalds * tcp_do_sendmsg to avoid burstiness. 2001da177e4SLinus Torvalds * Eric Schenk : Fix fast close down bug with 2011da177e4SLinus Torvalds * shutdown() followed by close(). 2021da177e4SLinus Torvalds * Andi Kleen : Make poll agree with SIGIO 2031da177e4SLinus Torvalds * Salvatore Sanfilippo : Support SO_LINGER with linger == 1 and 2041da177e4SLinus Torvalds * lingertime == 0 (RFC 793 ABORT Call) 2051da177e4SLinus Torvalds * Hirokazu Takahashi : Use copy_from_user() instead of 2061da177e4SLinus Torvalds * csum_and_copy_from_user() if possible. 2071da177e4SLinus Torvalds * 2081da177e4SLinus Torvalds * This program is free software; you can redistribute it and/or 2091da177e4SLinus Torvalds * modify it under the terms of the GNU General Public License 2101da177e4SLinus Torvalds * as published by the Free Software Foundation; either version 2111da177e4SLinus Torvalds * 2 of the License, or(at your option) any later version. 2121da177e4SLinus Torvalds * 2131da177e4SLinus Torvalds * Description of States: 2141da177e4SLinus Torvalds * 2151da177e4SLinus Torvalds * TCP_SYN_SENT sent a connection request, waiting for ack 2161da177e4SLinus Torvalds * 2171da177e4SLinus Torvalds * TCP_SYN_RECV received a connection request, sent ack, 2181da177e4SLinus Torvalds * waiting for final ack in three-way handshake. 2191da177e4SLinus Torvalds * 2201da177e4SLinus Torvalds * TCP_ESTABLISHED connection established 2211da177e4SLinus Torvalds * 2221da177e4SLinus Torvalds * TCP_FIN_WAIT1 our side has shutdown, waiting to complete 2231da177e4SLinus Torvalds * transmission of remaining buffered data 2241da177e4SLinus Torvalds * 2251da177e4SLinus Torvalds * TCP_FIN_WAIT2 all buffered data sent, waiting for remote 2261da177e4SLinus Torvalds * to shutdown 2271da177e4SLinus Torvalds * 2281da177e4SLinus Torvalds * TCP_CLOSING both sides have shutdown but we still have 2291da177e4SLinus Torvalds * data we have to finish sending 2301da177e4SLinus Torvalds * 2311da177e4SLinus Torvalds * TCP_TIME_WAIT timeout to catch resent junk before entering 2321da177e4SLinus Torvalds * closed, can only be entered from FIN_WAIT2 2331da177e4SLinus Torvalds * or CLOSING. Required because the other end 2341da177e4SLinus Torvalds * may not have gotten our last ACK causing it 2351da177e4SLinus Torvalds * to retransmit the data packet (which we ignore) 2361da177e4SLinus Torvalds * 2371da177e4SLinus Torvalds * TCP_CLOSE_WAIT remote side has shutdown and is waiting for 2381da177e4SLinus Torvalds * us to finish writing our data and to shutdown 2391da177e4SLinus Torvalds * (we have to close() to move on to LAST_ACK) 2401da177e4SLinus Torvalds * 2411da177e4SLinus Torvalds * TCP_LAST_ACK out side has shutdown after remote has 2421da177e4SLinus Torvalds * shutdown. There may still be data in our 2431da177e4SLinus Torvalds * buffer that we have to finish sending 2441da177e4SLinus Torvalds * 2451da177e4SLinus Torvalds * TCP_CLOSE socket is finished 2461da177e4SLinus Torvalds */ 2471da177e4SLinus Torvalds 248afd46503SJoe Perches #define pr_fmt(fmt) "TCP: " fmt 249afd46503SJoe Perches 250172589ccSIlpo Järvinen #include <linux/kernel.h> 2511da177e4SLinus Torvalds #include <linux/module.h> 2521da177e4SLinus Torvalds #include <linux/types.h> 2531da177e4SLinus Torvalds #include <linux/fcntl.h> 2541da177e4SLinus Torvalds #include <linux/poll.h> 2551da177e4SLinus Torvalds #include <linux/init.h> 2561da177e4SLinus Torvalds #include <linux/fs.h> 2579c55e01cSJens Axboe #include <linux/skbuff.h> 25881b23b4aSAndrew Morton #include <linux/scatterlist.h> 2599c55e01cSJens Axboe #include <linux/splice.h> 2609c55e01cSJens Axboe #include <linux/net.h> 2619c55e01cSJens Axboe #include <linux/socket.h> 2621da177e4SLinus Torvalds #include <linux/random.h> 2631da177e4SLinus Torvalds #include <linux/bootmem.h> 26457413ebcSMiquel van Smoorenburg #include <linux/highmem.h> 26557413ebcSMiquel van Smoorenburg #include <linux/swap.h> 266b8059eadSDavid S. Miller #include <linux/cache.h> 267f4c50d99SHerbert Xu #include <linux/err.h> 268cfb6eeb4SYOSHIFUJI Hideaki #include <linux/crypto.h> 269da5c78c8SWilliam Allen Simpson #include <linux/time.h> 2705a0e3ad6STejun Heo #include <linux/slab.h> 2711da177e4SLinus Torvalds 2721da177e4SLinus Torvalds #include <net/icmp.h> 273cf60af03SYuchung Cheng #include <net/inet_common.h> 2741da177e4SLinus Torvalds #include <net/tcp.h> 2751da177e4SLinus Torvalds #include <net/xfrm.h> 2761da177e4SLinus Torvalds #include <net/ip.h> 2771a2449a8SChris Leech #include <net/netdma.h> 2789c55e01cSJens Axboe #include <net/sock.h> 2791da177e4SLinus Torvalds 2801da177e4SLinus Torvalds #include <asm/uaccess.h> 2811da177e4SLinus Torvalds #include <asm/ioctls.h> 282076bb0c8SEliezer Tamir #include <net/busy_poll.h> 2831da177e4SLinus Torvalds 284ab32ea5dSBrian Haley int sysctl_tcp_fin_timeout __read_mostly = TCP_FIN_TIMEOUT; 2851da177e4SLinus Torvalds 286dd24c001SEric Dumazet struct percpu_counter tcp_orphan_count; 2870a5578cfSArnaldo Carvalho de Melo EXPORT_SYMBOL_GPL(tcp_orphan_count); 2880a5578cfSArnaldo Carvalho de Melo 289b8059eadSDavid S. Miller int sysctl_tcp_wmem[3] __read_mostly; 290b8059eadSDavid S. Miller int sysctl_tcp_rmem[3] __read_mostly; 2911da177e4SLinus Torvalds 2921da177e4SLinus Torvalds EXPORT_SYMBOL(sysctl_tcp_rmem); 2931da177e4SLinus Torvalds EXPORT_SYMBOL(sysctl_tcp_wmem); 2941da177e4SLinus Torvalds 2958d987e5cSEric Dumazet atomic_long_t tcp_memory_allocated; /* Current allocated memory. */ 2961da177e4SLinus Torvalds EXPORT_SYMBOL(tcp_memory_allocated); 2971748376bSEric Dumazet 2981748376bSEric Dumazet /* 2991748376bSEric Dumazet * Current number of TCP sockets. 3001748376bSEric Dumazet */ 3011748376bSEric Dumazet struct percpu_counter tcp_sockets_allocated; 3021da177e4SLinus Torvalds EXPORT_SYMBOL(tcp_sockets_allocated); 3031da177e4SLinus Torvalds 3041da177e4SLinus Torvalds /* 3059c55e01cSJens Axboe * TCP splice context 3069c55e01cSJens Axboe */ 3079c55e01cSJens Axboe struct tcp_splice_state { 3089c55e01cSJens Axboe struct pipe_inode_info *pipe; 3099c55e01cSJens Axboe size_t len; 3109c55e01cSJens Axboe unsigned int flags; 3119c55e01cSJens Axboe }; 3129c55e01cSJens Axboe 3139c55e01cSJens Axboe /* 3141da177e4SLinus Torvalds * Pressure flag: try to collapse. 3151da177e4SLinus Torvalds * Technical note: it is used by multiple contexts non atomically. 3163ab224beSHideo Aoki * All the __sk_mem_schedule() is of this nature: accounting 3171da177e4SLinus Torvalds * is strict, actions are advisory and have some latency. 3181da177e4SLinus Torvalds */ 3194103f8cdSEric Dumazet int tcp_memory_pressure __read_mostly; 3201da177e4SLinus Torvalds EXPORT_SYMBOL(tcp_memory_pressure); 3211da177e4SLinus Torvalds 3225c52ba17SPavel Emelyanov void tcp_enter_memory_pressure(struct sock *sk) 3231da177e4SLinus Torvalds { 3241da177e4SLinus Torvalds if (!tcp_memory_pressure) { 3254e673444SPavel Emelyanov NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPMEMORYPRESSURES); 3261da177e4SLinus Torvalds tcp_memory_pressure = 1; 3271da177e4SLinus Torvalds } 3281da177e4SLinus Torvalds } 3291da177e4SLinus Torvalds EXPORT_SYMBOL(tcp_enter_memory_pressure); 3301da177e4SLinus Torvalds 331b103cf34SJulian Anastasov /* Convert seconds to retransmits based on initial and max timeout */ 332b103cf34SJulian Anastasov static u8 secs_to_retrans(int seconds, int timeout, int rto_max) 333b103cf34SJulian Anastasov { 334b103cf34SJulian Anastasov u8 res = 0; 335b103cf34SJulian Anastasov 336b103cf34SJulian Anastasov if (seconds > 0) { 337b103cf34SJulian Anastasov int period = timeout; 338b103cf34SJulian Anastasov 339b103cf34SJulian Anastasov res = 1; 340b103cf34SJulian Anastasov while (seconds > period && res < 255) { 341b103cf34SJulian Anastasov res++; 342b103cf34SJulian Anastasov timeout <<= 1; 343b103cf34SJulian Anastasov if (timeout > rto_max) 344b103cf34SJulian Anastasov timeout = rto_max; 345b103cf34SJulian Anastasov period += timeout; 346b103cf34SJulian Anastasov } 347b103cf34SJulian Anastasov } 348b103cf34SJulian Anastasov return res; 349b103cf34SJulian Anastasov } 350b103cf34SJulian Anastasov 351b103cf34SJulian Anastasov /* Convert retransmits to seconds based on initial and max timeout */ 352b103cf34SJulian Anastasov static int retrans_to_secs(u8 retrans, int timeout, int rto_max) 353b103cf34SJulian Anastasov { 354b103cf34SJulian Anastasov int period = 0; 355b103cf34SJulian Anastasov 356b103cf34SJulian Anastasov if (retrans > 0) { 357b103cf34SJulian Anastasov period = timeout; 358b103cf34SJulian Anastasov while (--retrans) { 359b103cf34SJulian Anastasov timeout <<= 1; 360b103cf34SJulian Anastasov if (timeout > rto_max) 361b103cf34SJulian Anastasov timeout = rto_max; 362b103cf34SJulian Anastasov period += timeout; 363b103cf34SJulian Anastasov } 364b103cf34SJulian Anastasov } 365b103cf34SJulian Anastasov return period; 366b103cf34SJulian Anastasov } 367b103cf34SJulian Anastasov 368900f65d3SNeal Cardwell /* Address-family independent initialization for a tcp_sock. 369900f65d3SNeal Cardwell * 370900f65d3SNeal Cardwell * NOTE: A lot of things set to zero explicitly by call to 371900f65d3SNeal Cardwell * sk_alloc() so need not be done here. 372900f65d3SNeal Cardwell */ 373900f65d3SNeal Cardwell void tcp_init_sock(struct sock *sk) 374900f65d3SNeal Cardwell { 375900f65d3SNeal Cardwell struct inet_connection_sock *icsk = inet_csk(sk); 376900f65d3SNeal Cardwell struct tcp_sock *tp = tcp_sk(sk); 377900f65d3SNeal Cardwell 378900f65d3SNeal Cardwell skb_queue_head_init(&tp->out_of_order_queue); 379900f65d3SNeal Cardwell tcp_init_xmit_timers(sk); 380900f65d3SNeal Cardwell tcp_prequeue_init(tp); 38146d3ceabSEric Dumazet INIT_LIST_HEAD(&tp->tsq_node); 382900f65d3SNeal Cardwell 383900f65d3SNeal Cardwell icsk->icsk_rto = TCP_TIMEOUT_INIT; 384900f65d3SNeal Cardwell tp->mdev = TCP_TIMEOUT_INIT; 385900f65d3SNeal Cardwell 386900f65d3SNeal Cardwell /* So many TCP implementations out there (incorrectly) count the 387900f65d3SNeal Cardwell * initial SYN frame in their delayed-ACK and congestion control 388900f65d3SNeal Cardwell * algorithms that we must have the following bandaid to talk 389900f65d3SNeal Cardwell * efficiently to them. -DaveM 390900f65d3SNeal Cardwell */ 391900f65d3SNeal Cardwell tp->snd_cwnd = TCP_INIT_CWND; 392900f65d3SNeal Cardwell 393900f65d3SNeal Cardwell /* See draft-stevens-tcpca-spec-01 for discussion of the 394900f65d3SNeal Cardwell * initialization of these values. 395900f65d3SNeal Cardwell */ 396900f65d3SNeal Cardwell tp->snd_ssthresh = TCP_INFINITE_SSTHRESH; 397900f65d3SNeal Cardwell tp->snd_cwnd_clamp = ~0; 398900f65d3SNeal Cardwell tp->mss_cache = TCP_MSS_DEFAULT; 399900f65d3SNeal Cardwell 400900f65d3SNeal Cardwell tp->reordering = sysctl_tcp_reordering; 401eed530b6SYuchung Cheng tcp_enable_early_retrans(tp); 402900f65d3SNeal Cardwell icsk->icsk_ca_ops = &tcp_init_congestion_ops; 403900f65d3SNeal Cardwell 404ceaa1fefSAndrey Vagin tp->tsoffset = 0; 405ceaa1fefSAndrey Vagin 406900f65d3SNeal Cardwell sk->sk_state = TCP_CLOSE; 407900f65d3SNeal Cardwell 408900f65d3SNeal Cardwell sk->sk_write_space = sk_stream_write_space; 409900f65d3SNeal Cardwell sock_set_flag(sk, SOCK_USE_WRITE_QUEUE); 410900f65d3SNeal Cardwell 411900f65d3SNeal Cardwell icsk->icsk_sync_mss = tcp_sync_mss; 412900f65d3SNeal Cardwell 413900f65d3SNeal Cardwell /* Presumed zeroed, in order of appearance: 414900f65d3SNeal Cardwell * cookie_in_always, cookie_out_never, 415900f65d3SNeal Cardwell * s_data_constant, s_data_in, s_data_out 416900f65d3SNeal Cardwell */ 417900f65d3SNeal Cardwell sk->sk_sndbuf = sysctl_tcp_wmem[1]; 418900f65d3SNeal Cardwell sk->sk_rcvbuf = sysctl_tcp_rmem[1]; 419900f65d3SNeal Cardwell 420900f65d3SNeal Cardwell local_bh_disable(); 421900f65d3SNeal Cardwell sock_update_memcg(sk); 422900f65d3SNeal Cardwell sk_sockets_allocated_inc(sk); 423900f65d3SNeal Cardwell local_bh_enable(); 424900f65d3SNeal Cardwell } 425900f65d3SNeal Cardwell EXPORT_SYMBOL(tcp_init_sock); 426900f65d3SNeal Cardwell 4271da177e4SLinus Torvalds /* 4281da177e4SLinus Torvalds * Wait for a TCP event. 4291da177e4SLinus Torvalds * 4301da177e4SLinus Torvalds * Note that we don't need to lock the socket, as the upper poll layers 4311da177e4SLinus Torvalds * take care of normal races (between the test and the event) and we don't 4321da177e4SLinus Torvalds * go look at any of the socket buffers directly. 4331da177e4SLinus Torvalds */ 4341da177e4SLinus Torvalds unsigned int tcp_poll(struct file *file, struct socket *sock, poll_table *wait) 4351da177e4SLinus Torvalds { 4361da177e4SLinus Torvalds unsigned int mask; 4371da177e4SLinus Torvalds struct sock *sk = sock->sk; 438cf533ea5SEric Dumazet const struct tcp_sock *tp = tcp_sk(sk); 4391da177e4SLinus Torvalds 440c3f1dbafSDavid Majnemer sock_rps_record_flow(sk); 441c3f1dbafSDavid Majnemer 442aa395145SEric Dumazet sock_poll_wait(file, sk_sleep(sk), wait); 4431da177e4SLinus Torvalds if (sk->sk_state == TCP_LISTEN) 444dc40c7bcSArnaldo Carvalho de Melo return inet_csk_listen_poll(sk); 4451da177e4SLinus Torvalds 4461da177e4SLinus Torvalds /* Socket is not locked. We are protected from async events 44770efce27SWill Newton * by poll logic and correct handling of state changes 44870efce27SWill Newton * made by other threads is impossible in any case. 4491da177e4SLinus Torvalds */ 4501da177e4SLinus Torvalds 4511da177e4SLinus Torvalds mask = 0; 4521da177e4SLinus Torvalds 4531da177e4SLinus Torvalds /* 4541da177e4SLinus Torvalds * POLLHUP is certainly not done right. But poll() doesn't 4551da177e4SLinus Torvalds * have a notion of HUP in just one direction, and for a 4561da177e4SLinus Torvalds * socket the read side is more interesting. 4571da177e4SLinus Torvalds * 4581da177e4SLinus Torvalds * Some poll() documentation says that POLLHUP is incompatible 4591da177e4SLinus Torvalds * with the POLLOUT/POLLWR flags, so somebody should check this 4601da177e4SLinus Torvalds * all. But careful, it tends to be safer to return too many 4611da177e4SLinus Torvalds * bits than too few, and you can easily break real applications 4621da177e4SLinus Torvalds * if you don't tell them that something has hung up! 4631da177e4SLinus Torvalds * 4641da177e4SLinus Torvalds * Check-me. 4651da177e4SLinus Torvalds * 4661da177e4SLinus Torvalds * Check number 1. POLLHUP is _UNMASKABLE_ event (see UNIX98 and 4671da177e4SLinus Torvalds * our fs/select.c). It means that after we received EOF, 4681da177e4SLinus Torvalds * poll always returns immediately, making impossible poll() on write() 4691da177e4SLinus Torvalds * in state CLOSE_WAIT. One solution is evident --- to set POLLHUP 4701da177e4SLinus Torvalds * if and only if shutdown has been made in both directions. 4711da177e4SLinus Torvalds * Actually, it is interesting to look how Solaris and DUX 47270efce27SWill Newton * solve this dilemma. I would prefer, if POLLHUP were maskable, 4731da177e4SLinus Torvalds * then we could set it on SND_SHUTDOWN. BTW examples given 4741da177e4SLinus Torvalds * in Stevens' books assume exactly this behaviour, it explains 47570efce27SWill Newton * why POLLHUP is incompatible with POLLOUT. --ANK 4761da177e4SLinus Torvalds * 4771da177e4SLinus Torvalds * NOTE. Check for TCP_CLOSE is added. The goal is to prevent 4781da177e4SLinus Torvalds * blocking on fresh not-connected or disconnected socket. --ANK 4791da177e4SLinus Torvalds */ 4801da177e4SLinus Torvalds if (sk->sk_shutdown == SHUTDOWN_MASK || sk->sk_state == TCP_CLOSE) 4811da177e4SLinus Torvalds mask |= POLLHUP; 4821da177e4SLinus Torvalds if (sk->sk_shutdown & RCV_SHUTDOWN) 483f348d70aSDavide Libenzi mask |= POLLIN | POLLRDNORM | POLLRDHUP; 4841da177e4SLinus Torvalds 4858336886fSJerry Chu /* Connected or passive Fast Open socket? */ 4868336886fSJerry Chu if (sk->sk_state != TCP_SYN_SENT && 4878336886fSJerry Chu (sk->sk_state != TCP_SYN_RECV || tp->fastopen_rsk != NULL)) { 488c7004482SDavid S. Miller int target = sock_rcvlowat(sk, 0, INT_MAX); 489c7004482SDavid S. Miller 490c7004482SDavid S. Miller if (tp->urg_seq == tp->copied_seq && 491c7004482SDavid S. Miller !sock_flag(sk, SOCK_URGINLINE) && 492c7004482SDavid S. Miller tp->urg_data) 493b634f875SAlexandra Kossovsky target++; 494c7004482SDavid S. Miller 4951da177e4SLinus Torvalds /* Potential race condition. If read of tp below will 4961da177e4SLinus Torvalds * escape above sk->sk_state, we can be illegally awaken 4971da177e4SLinus Torvalds * in SYN_* states. */ 498c7004482SDavid S. Miller if (tp->rcv_nxt - tp->copied_seq >= target) 4991da177e4SLinus Torvalds mask |= POLLIN | POLLRDNORM; 5001da177e4SLinus Torvalds 5011da177e4SLinus Torvalds if (!(sk->sk_shutdown & SEND_SHUTDOWN)) { 502*64dc6130SEric Dumazet if (sk_stream_is_writeable(sk)) { 5031da177e4SLinus Torvalds mask |= POLLOUT | POLLWRNORM; 5041da177e4SLinus Torvalds } else { /* send SIGIO later */ 5051da177e4SLinus Torvalds set_bit(SOCK_ASYNC_NOSPACE, 5061da177e4SLinus Torvalds &sk->sk_socket->flags); 5071da177e4SLinus Torvalds set_bit(SOCK_NOSPACE, &sk->sk_socket->flags); 5081da177e4SLinus Torvalds 5091da177e4SLinus Torvalds /* Race breaker. If space is freed after 5101da177e4SLinus Torvalds * wspace test but before the flags are set, 5111da177e4SLinus Torvalds * IO signal will be lost. 5121da177e4SLinus Torvalds */ 513*64dc6130SEric Dumazet if (sk_stream_is_writeable(sk)) 5141da177e4SLinus Torvalds mask |= POLLOUT | POLLWRNORM; 5151da177e4SLinus Torvalds } 516d84ba638SKOSAKI Motohiro } else 517d84ba638SKOSAKI Motohiro mask |= POLLOUT | POLLWRNORM; 5181da177e4SLinus Torvalds 5191da177e4SLinus Torvalds if (tp->urg_data & TCP_URG_VALID) 5201da177e4SLinus Torvalds mask |= POLLPRI; 5211da177e4SLinus Torvalds } 522a4d25803STom Marshall /* This barrier is coupled with smp_wmb() in tcp_reset() */ 523a4d25803STom Marshall smp_rmb(); 524a4d25803STom Marshall if (sk->sk_err) 525a4d25803STom Marshall mask |= POLLERR; 526a4d25803STom Marshall 5271da177e4SLinus Torvalds return mask; 5281da177e4SLinus Torvalds } 5294bc2f18bSEric Dumazet EXPORT_SYMBOL(tcp_poll); 5301da177e4SLinus Torvalds 5311da177e4SLinus Torvalds int tcp_ioctl(struct sock *sk, int cmd, unsigned long arg) 5321da177e4SLinus Torvalds { 5331da177e4SLinus Torvalds struct tcp_sock *tp = tcp_sk(sk); 5341da177e4SLinus Torvalds int answ; 5350e71c55cSEric Dumazet bool slow; 5361da177e4SLinus Torvalds 5371da177e4SLinus Torvalds switch (cmd) { 5381da177e4SLinus Torvalds case SIOCINQ: 5391da177e4SLinus Torvalds if (sk->sk_state == TCP_LISTEN) 5401da177e4SLinus Torvalds return -EINVAL; 5411da177e4SLinus Torvalds 5420e71c55cSEric Dumazet slow = lock_sock_fast(sk); 5431da177e4SLinus Torvalds if ((1 << sk->sk_state) & (TCPF_SYN_SENT | TCPF_SYN_RECV)) 5441da177e4SLinus Torvalds answ = 0; 5451da177e4SLinus Torvalds else if (sock_flag(sk, SOCK_URGINLINE) || 5461da177e4SLinus Torvalds !tp->urg_data || 5471da177e4SLinus Torvalds before(tp->urg_seq, tp->copied_seq) || 5481da177e4SLinus Torvalds !before(tp->urg_seq, tp->rcv_nxt)) { 54991521944SDavid S. Miller 5501da177e4SLinus Torvalds answ = tp->rcv_nxt - tp->copied_seq; 5511da177e4SLinus Torvalds 552a3374c42SEric Dumazet /* Subtract 1, if FIN was received */ 553a3374c42SEric Dumazet if (answ && sock_flag(sk, SOCK_DONE)) 554a3374c42SEric Dumazet answ--; 5551da177e4SLinus Torvalds } else 5561da177e4SLinus Torvalds answ = tp->urg_seq - tp->copied_seq; 5570e71c55cSEric Dumazet unlock_sock_fast(sk, slow); 5581da177e4SLinus Torvalds break; 5591da177e4SLinus Torvalds case SIOCATMARK: 5601da177e4SLinus Torvalds answ = tp->urg_data && tp->urg_seq == tp->copied_seq; 5611da177e4SLinus Torvalds break; 5621da177e4SLinus Torvalds case SIOCOUTQ: 5631da177e4SLinus Torvalds if (sk->sk_state == TCP_LISTEN) 5641da177e4SLinus Torvalds return -EINVAL; 5651da177e4SLinus Torvalds 5661da177e4SLinus Torvalds if ((1 << sk->sk_state) & (TCPF_SYN_SENT | TCPF_SYN_RECV)) 5671da177e4SLinus Torvalds answ = 0; 5681da177e4SLinus Torvalds else 5691da177e4SLinus Torvalds answ = tp->write_seq - tp->snd_una; 5701da177e4SLinus Torvalds break; 5712f4e1b39SMario Schuknecht case SIOCOUTQNSD: 5722f4e1b39SMario Schuknecht if (sk->sk_state == TCP_LISTEN) 5732f4e1b39SMario Schuknecht return -EINVAL; 5742f4e1b39SMario Schuknecht 5752f4e1b39SMario Schuknecht if ((1 << sk->sk_state) & (TCPF_SYN_SENT | TCPF_SYN_RECV)) 5762f4e1b39SMario Schuknecht answ = 0; 5772f4e1b39SMario Schuknecht else 5782f4e1b39SMario Schuknecht answ = tp->write_seq - tp->snd_nxt; 5792f4e1b39SMario Schuknecht break; 5801da177e4SLinus Torvalds default: 5811da177e4SLinus Torvalds return -ENOIOCTLCMD; 5823ff50b79SStephen Hemminger } 5831da177e4SLinus Torvalds 5841da177e4SLinus Torvalds return put_user(answ, (int __user *)arg); 5851da177e4SLinus Torvalds } 5864bc2f18bSEric Dumazet EXPORT_SYMBOL(tcp_ioctl); 5871da177e4SLinus Torvalds 5881da177e4SLinus Torvalds static inline void tcp_mark_push(struct tcp_sock *tp, struct sk_buff *skb) 5891da177e4SLinus Torvalds { 5904de075e0SEric Dumazet TCP_SKB_CB(skb)->tcp_flags |= TCPHDR_PSH; 5911da177e4SLinus Torvalds tp->pushed_seq = tp->write_seq; 5921da177e4SLinus Torvalds } 5931da177e4SLinus Torvalds 594a2a385d6SEric Dumazet static inline bool forced_push(const struct tcp_sock *tp) 5951da177e4SLinus Torvalds { 5961da177e4SLinus Torvalds return after(tp->write_seq, tp->pushed_seq + (tp->max_window >> 1)); 5971da177e4SLinus Torvalds } 5981da177e4SLinus Torvalds 5999e412ba7SIlpo Järvinen static inline void skb_entail(struct sock *sk, struct sk_buff *skb) 6001da177e4SLinus Torvalds { 6019e412ba7SIlpo Järvinen struct tcp_sock *tp = tcp_sk(sk); 602352d4800SArnaldo Carvalho de Melo struct tcp_skb_cb *tcb = TCP_SKB_CB(skb); 603352d4800SArnaldo Carvalho de Melo 6041da177e4SLinus Torvalds skb->csum = 0; 605352d4800SArnaldo Carvalho de Melo tcb->seq = tcb->end_seq = tp->write_seq; 6064de075e0SEric Dumazet tcb->tcp_flags = TCPHDR_ACK; 607352d4800SArnaldo Carvalho de Melo tcb->sacked = 0; 6081da177e4SLinus Torvalds skb_header_release(skb); 609fe067e8aSDavid S. Miller tcp_add_write_queue_tail(sk, skb); 6103ab224beSHideo Aoki sk->sk_wmem_queued += skb->truesize; 6113ab224beSHideo Aoki sk_mem_charge(sk, skb->truesize); 61289ebd197SDavid S. Miller if (tp->nonagle & TCP_NAGLE_PUSH) 6131da177e4SLinus Torvalds tp->nonagle &= ~TCP_NAGLE_PUSH; 6141da177e4SLinus Torvalds } 6151da177e4SLinus Torvalds 616afeca340SKrishna Kumar static inline void tcp_mark_urg(struct tcp_sock *tp, int flags) 6171da177e4SLinus Torvalds { 61833f5f57eSIlpo Järvinen if (flags & MSG_OOB) 6191da177e4SLinus Torvalds tp->snd_up = tp->write_seq; 6201da177e4SLinus Torvalds } 6211da177e4SLinus Torvalds 6229e412ba7SIlpo Järvinen static inline void tcp_push(struct sock *sk, int flags, int mss_now, 6239e412ba7SIlpo Järvinen int nonagle) 6241da177e4SLinus Torvalds { 625afeca340SKrishna Kumar if (tcp_send_head(sk)) { 6269e412ba7SIlpo Järvinen struct tcp_sock *tp = tcp_sk(sk); 6279e412ba7SIlpo Järvinen 6281da177e4SLinus Torvalds if (!(flags & MSG_MORE) || forced_push(tp)) 629afeca340SKrishna Kumar tcp_mark_push(tp, tcp_write_queue_tail(sk)); 630afeca340SKrishna Kumar 631afeca340SKrishna Kumar tcp_mark_urg(tp, flags); 6329e412ba7SIlpo Järvinen __tcp_push_pending_frames(sk, mss_now, 6331da177e4SLinus Torvalds (flags & MSG_MORE) ? TCP_NAGLE_CORK : nonagle); 6341da177e4SLinus Torvalds } 6351da177e4SLinus Torvalds } 6361da177e4SLinus Torvalds 6376ff7751dSAdrian Bunk static int tcp_splice_data_recv(read_descriptor_t *rd_desc, struct sk_buff *skb, 6389c55e01cSJens Axboe unsigned int offset, size_t len) 6399c55e01cSJens Axboe { 6409c55e01cSJens Axboe struct tcp_splice_state *tss = rd_desc->arg.data; 64133966dd0SWilly Tarreau int ret; 6429c55e01cSJens Axboe 6439fa5fdf2SDimitris Michailidis ret = skb_splice_bits(skb, offset, tss->pipe, min(rd_desc->count, len), 6449fa5fdf2SDimitris Michailidis tss->flags); 64533966dd0SWilly Tarreau if (ret > 0) 64633966dd0SWilly Tarreau rd_desc->count -= ret; 64733966dd0SWilly Tarreau return ret; 6489c55e01cSJens Axboe } 6499c55e01cSJens Axboe 6509c55e01cSJens Axboe static int __tcp_splice_read(struct sock *sk, struct tcp_splice_state *tss) 6519c55e01cSJens Axboe { 6529c55e01cSJens Axboe /* Store TCP splice context information in read_descriptor_t. */ 6539c55e01cSJens Axboe read_descriptor_t rd_desc = { 6549c55e01cSJens Axboe .arg.data = tss, 65533966dd0SWilly Tarreau .count = tss->len, 6569c55e01cSJens Axboe }; 6579c55e01cSJens Axboe 6589c55e01cSJens Axboe return tcp_read_sock(sk, &rd_desc, tcp_splice_data_recv); 6599c55e01cSJens Axboe } 6609c55e01cSJens Axboe 6619c55e01cSJens Axboe /** 6629c55e01cSJens Axboe * tcp_splice_read - splice data from TCP socket to a pipe 6639c55e01cSJens Axboe * @sock: socket to splice from 6649c55e01cSJens Axboe * @ppos: position (not valid) 6659c55e01cSJens Axboe * @pipe: pipe to splice to 6669c55e01cSJens Axboe * @len: number of bytes to splice 6679c55e01cSJens Axboe * @flags: splice modifier flags 6689c55e01cSJens Axboe * 6699c55e01cSJens Axboe * Description: 6709c55e01cSJens Axboe * Will read pages from given socket and fill them into a pipe. 6719c55e01cSJens Axboe * 6729c55e01cSJens Axboe **/ 6739c55e01cSJens Axboe ssize_t tcp_splice_read(struct socket *sock, loff_t *ppos, 6749c55e01cSJens Axboe struct pipe_inode_info *pipe, size_t len, 6759c55e01cSJens Axboe unsigned int flags) 6769c55e01cSJens Axboe { 6779c55e01cSJens Axboe struct sock *sk = sock->sk; 6789c55e01cSJens Axboe struct tcp_splice_state tss = { 6799c55e01cSJens Axboe .pipe = pipe, 6809c55e01cSJens Axboe .len = len, 6819c55e01cSJens Axboe .flags = flags, 6829c55e01cSJens Axboe }; 6839c55e01cSJens Axboe long timeo; 6849c55e01cSJens Axboe ssize_t spliced; 6859c55e01cSJens Axboe int ret; 6869c55e01cSJens Axboe 6873a047bf8SChangli Gao sock_rps_record_flow(sk); 6889c55e01cSJens Axboe /* 6899c55e01cSJens Axboe * We can't seek on a socket input 6909c55e01cSJens Axboe */ 6919c55e01cSJens Axboe if (unlikely(*ppos)) 6929c55e01cSJens Axboe return -ESPIPE; 6939c55e01cSJens Axboe 6949c55e01cSJens Axboe ret = spliced = 0; 6959c55e01cSJens Axboe 6969c55e01cSJens Axboe lock_sock(sk); 6979c55e01cSJens Axboe 69842324c62SEric Dumazet timeo = sock_rcvtimeo(sk, sock->file->f_flags & O_NONBLOCK); 6999c55e01cSJens Axboe while (tss.len) { 7009c55e01cSJens Axboe ret = __tcp_splice_read(sk, &tss); 7019c55e01cSJens Axboe if (ret < 0) 7029c55e01cSJens Axboe break; 7039c55e01cSJens Axboe else if (!ret) { 7049c55e01cSJens Axboe if (spliced) 7059c55e01cSJens Axboe break; 7069c55e01cSJens Axboe if (sock_flag(sk, SOCK_DONE)) 7079c55e01cSJens Axboe break; 7089c55e01cSJens Axboe if (sk->sk_err) { 7099c55e01cSJens Axboe ret = sock_error(sk); 7109c55e01cSJens Axboe break; 7119c55e01cSJens Axboe } 7129c55e01cSJens Axboe if (sk->sk_shutdown & RCV_SHUTDOWN) 7139c55e01cSJens Axboe break; 7149c55e01cSJens Axboe if (sk->sk_state == TCP_CLOSE) { 7159c55e01cSJens Axboe /* 7169c55e01cSJens Axboe * This occurs when user tries to read 7179c55e01cSJens Axboe * from never connected socket. 7189c55e01cSJens Axboe */ 7199c55e01cSJens Axboe if (!sock_flag(sk, SOCK_DONE)) 7209c55e01cSJens Axboe ret = -ENOTCONN; 7219c55e01cSJens Axboe break; 7229c55e01cSJens Axboe } 7239c55e01cSJens Axboe if (!timeo) { 7249c55e01cSJens Axboe ret = -EAGAIN; 7259c55e01cSJens Axboe break; 7269c55e01cSJens Axboe } 7279c55e01cSJens Axboe sk_wait_data(sk, &timeo); 7289c55e01cSJens Axboe if (signal_pending(current)) { 7299c55e01cSJens Axboe ret = sock_intr_errno(timeo); 7309c55e01cSJens Axboe break; 7319c55e01cSJens Axboe } 7329c55e01cSJens Axboe continue; 7339c55e01cSJens Axboe } 7349c55e01cSJens Axboe tss.len -= ret; 7359c55e01cSJens Axboe spliced += ret; 7369c55e01cSJens Axboe 73733966dd0SWilly Tarreau if (!timeo) 73833966dd0SWilly Tarreau break; 7399c55e01cSJens Axboe release_sock(sk); 7409c55e01cSJens Axboe lock_sock(sk); 7419c55e01cSJens Axboe 7429c55e01cSJens Axboe if (sk->sk_err || sk->sk_state == TCP_CLOSE || 74333966dd0SWilly Tarreau (sk->sk_shutdown & RCV_SHUTDOWN) || 7449c55e01cSJens Axboe signal_pending(current)) 7459c55e01cSJens Axboe break; 7469c55e01cSJens Axboe } 7479c55e01cSJens Axboe 7489c55e01cSJens Axboe release_sock(sk); 7499c55e01cSJens Axboe 7509c55e01cSJens Axboe if (spliced) 7519c55e01cSJens Axboe return spliced; 7529c55e01cSJens Axboe 7539c55e01cSJens Axboe return ret; 7549c55e01cSJens Axboe } 7554bc2f18bSEric Dumazet EXPORT_SYMBOL(tcp_splice_read); 7569c55e01cSJens Axboe 757df97c708SPavel Emelyanov struct sk_buff *sk_stream_alloc_skb(struct sock *sk, int size, gfp_t gfp) 758f561d0f2SPavel Emelyanov { 759f561d0f2SPavel Emelyanov struct sk_buff *skb; 760f561d0f2SPavel Emelyanov 761f561d0f2SPavel Emelyanov /* The TCP header must be at least 32-bit aligned. */ 762f561d0f2SPavel Emelyanov size = ALIGN(size, 4); 763f561d0f2SPavel Emelyanov 764f561d0f2SPavel Emelyanov skb = alloc_skb_fclone(size + sk->sk_prot->max_header, gfp); 765f561d0f2SPavel Emelyanov if (skb) { 7663ab224beSHideo Aoki if (sk_wmem_schedule(sk, skb->truesize)) { 767a21d4572SEric Dumazet skb_reserve(skb, sk->sk_prot->max_header); 768f561d0f2SPavel Emelyanov /* 769f561d0f2SPavel Emelyanov * Make sure that we have exactly size bytes 770f561d0f2SPavel Emelyanov * available to the caller, no more, no less. 771f561d0f2SPavel Emelyanov */ 77216fad69cSEric Dumazet skb->reserved_tailroom = skb->end - skb->tail - size; 773f561d0f2SPavel Emelyanov return skb; 774f561d0f2SPavel Emelyanov } 775f561d0f2SPavel Emelyanov __kfree_skb(skb); 776f561d0f2SPavel Emelyanov } else { 7775c52ba17SPavel Emelyanov sk->sk_prot->enter_memory_pressure(sk); 778f561d0f2SPavel Emelyanov sk_stream_moderate_sndbuf(sk); 779f561d0f2SPavel Emelyanov } 780f561d0f2SPavel Emelyanov return NULL; 781f561d0f2SPavel Emelyanov } 782f561d0f2SPavel Emelyanov 7830c54b85fSIlpo Järvinen static unsigned int tcp_xmit_size_goal(struct sock *sk, u32 mss_now, 7840c54b85fSIlpo Järvinen int large_allowed) 7850c54b85fSIlpo Järvinen { 7860c54b85fSIlpo Järvinen struct tcp_sock *tp = tcp_sk(sk); 7872a3a041cSIlpo Järvinen u32 xmit_size_goal, old_size_goal; 7880c54b85fSIlpo Järvinen 7890c54b85fSIlpo Järvinen xmit_size_goal = mss_now; 7900c54b85fSIlpo Järvinen 7910c54b85fSIlpo Järvinen if (large_allowed && sk_can_gso(sk)) { 7920c54b85fSIlpo Järvinen xmit_size_goal = ((sk->sk_gso_max_size - 1) - 7930c54b85fSIlpo Järvinen inet_csk(sk)->icsk_af_ops->net_header_len - 7940c54b85fSIlpo Järvinen inet_csk(sk)->icsk_ext_hdr_len - 7950c54b85fSIlpo Järvinen tp->tcp_header_len); 7960c54b85fSIlpo Järvinen 79746d3ceabSEric Dumazet /* TSQ : try to have two TSO segments in flight */ 79846d3ceabSEric Dumazet xmit_size_goal = min_t(u32, xmit_size_goal, 79946d3ceabSEric Dumazet sysctl_tcp_limit_output_bytes >> 1); 80046d3ceabSEric Dumazet 8010c54b85fSIlpo Järvinen xmit_size_goal = tcp_bound_to_half_wnd(tp, xmit_size_goal); 8022a3a041cSIlpo Järvinen 8032a3a041cSIlpo Järvinen /* We try hard to avoid divides here */ 8042a3a041cSIlpo Järvinen old_size_goal = tp->xmit_size_goal_segs * mss_now; 8052a3a041cSIlpo Järvinen 8062a3a041cSIlpo Järvinen if (likely(old_size_goal <= xmit_size_goal && 8072a3a041cSIlpo Järvinen old_size_goal + mss_now > xmit_size_goal)) { 8082a3a041cSIlpo Järvinen xmit_size_goal = old_size_goal; 8092a3a041cSIlpo Järvinen } else { 8101485348dSBen Hutchings tp->xmit_size_goal_segs = 8111485348dSBen Hutchings min_t(u16, xmit_size_goal / mss_now, 8121485348dSBen Hutchings sk->sk_gso_max_segs); 8132a3a041cSIlpo Järvinen xmit_size_goal = tp->xmit_size_goal_segs * mss_now; 8142a3a041cSIlpo Järvinen } 8150c54b85fSIlpo Järvinen } 8160c54b85fSIlpo Järvinen 817afece1c6SIlpo Järvinen return max(xmit_size_goal, mss_now); 8180c54b85fSIlpo Järvinen } 8190c54b85fSIlpo Järvinen 8200c54b85fSIlpo Järvinen static int tcp_send_mss(struct sock *sk, int *size_goal, int flags) 8210c54b85fSIlpo Järvinen { 8220c54b85fSIlpo Järvinen int mss_now; 8230c54b85fSIlpo Järvinen 8240c54b85fSIlpo Järvinen mss_now = tcp_current_mss(sk); 8250c54b85fSIlpo Järvinen *size_goal = tcp_xmit_size_goal(sk, mss_now, !(flags & MSG_OOB)); 8260c54b85fSIlpo Järvinen 8270c54b85fSIlpo Järvinen return mss_now; 8280c54b85fSIlpo Järvinen } 8290c54b85fSIlpo Järvinen 83064022d0bSEric Dumazet static ssize_t do_tcp_sendpages(struct sock *sk, struct page *page, int offset, 83164022d0bSEric Dumazet size_t size, int flags) 8321da177e4SLinus Torvalds { 8331da177e4SLinus Torvalds struct tcp_sock *tp = tcp_sk(sk); 834c1b4a7e6SDavid S. Miller int mss_now, size_goal; 8351da177e4SLinus Torvalds int err; 8361da177e4SLinus Torvalds ssize_t copied; 8371da177e4SLinus Torvalds long timeo = sock_sndtimeo(sk, flags & MSG_DONTWAIT); 8381da177e4SLinus Torvalds 8398336886fSJerry Chu /* Wait for a connection to finish. One exception is TCP Fast Open 8408336886fSJerry Chu * (passive side) where data is allowed to be sent before a connection 8418336886fSJerry Chu * is fully established. 8428336886fSJerry Chu */ 8438336886fSJerry Chu if (((1 << sk->sk_state) & ~(TCPF_ESTABLISHED | TCPF_CLOSE_WAIT)) && 8448336886fSJerry Chu !tcp_passive_fastopen(sk)) { 8451da177e4SLinus Torvalds if ((err = sk_stream_wait_connect(sk, &timeo)) != 0) 8461da177e4SLinus Torvalds goto out_err; 8478336886fSJerry Chu } 8481da177e4SLinus Torvalds 8491da177e4SLinus Torvalds clear_bit(SOCK_ASYNC_NOSPACE, &sk->sk_socket->flags); 8501da177e4SLinus Torvalds 8510c54b85fSIlpo Järvinen mss_now = tcp_send_mss(sk, &size_goal, flags); 8521da177e4SLinus Torvalds copied = 0; 8531da177e4SLinus Torvalds 8541da177e4SLinus Torvalds err = -EPIPE; 8551da177e4SLinus Torvalds if (sk->sk_err || (sk->sk_shutdown & SEND_SHUTDOWN)) 8560d6a775eSIlpo Järvinen goto out_err; 8571da177e4SLinus Torvalds 85864022d0bSEric Dumazet while (size > 0) { 859fe067e8aSDavid S. Miller struct sk_buff *skb = tcp_write_queue_tail(sk); 86038ba0a65SEric Dumazet int copy, i; 86138ba0a65SEric Dumazet bool can_coalesce; 8621da177e4SLinus Torvalds 863fe067e8aSDavid S. Miller if (!tcp_send_head(sk) || (copy = size_goal - skb->len) <= 0) { 8641da177e4SLinus Torvalds new_segment: 8651da177e4SLinus Torvalds if (!sk_stream_memory_free(sk)) 8661da177e4SLinus Torvalds goto wait_for_sndbuf; 8671da177e4SLinus Torvalds 868df97c708SPavel Emelyanov skb = sk_stream_alloc_skb(sk, 0, sk->sk_allocation); 8691da177e4SLinus Torvalds if (!skb) 8701da177e4SLinus Torvalds goto wait_for_memory; 8711da177e4SLinus Torvalds 8729e412ba7SIlpo Järvinen skb_entail(sk, skb); 873c1b4a7e6SDavid S. Miller copy = size_goal; 8741da177e4SLinus Torvalds } 8751da177e4SLinus Torvalds 8761da177e4SLinus Torvalds if (copy > size) 8771da177e4SLinus Torvalds copy = size; 8781da177e4SLinus Torvalds 8791da177e4SLinus Torvalds i = skb_shinfo(skb)->nr_frags; 8801da177e4SLinus Torvalds can_coalesce = skb_can_coalesce(skb, i, page, offset); 8811da177e4SLinus Torvalds if (!can_coalesce && i >= MAX_SKB_FRAGS) { 8821da177e4SLinus Torvalds tcp_mark_push(tp, skb); 8831da177e4SLinus Torvalds goto new_segment; 8841da177e4SLinus Torvalds } 8853ab224beSHideo Aoki if (!sk_wmem_schedule(sk, copy)) 8861da177e4SLinus Torvalds goto wait_for_memory; 8871da177e4SLinus Torvalds 8881da177e4SLinus Torvalds if (can_coalesce) { 8899e903e08SEric Dumazet skb_frag_size_add(&skb_shinfo(skb)->frags[i - 1], copy); 8901da177e4SLinus Torvalds } else { 8911da177e4SLinus Torvalds get_page(page); 8921da177e4SLinus Torvalds skb_fill_page_desc(skb, i, page, offset, copy); 8931da177e4SLinus Torvalds } 894c9af6db4SPravin B Shelar skb_shinfo(skb)->tx_flags |= SKBTX_SHARED_FRAG; 895cef401deSEric Dumazet 8961da177e4SLinus Torvalds skb->len += copy; 8971da177e4SLinus Torvalds skb->data_len += copy; 8981da177e4SLinus Torvalds skb->truesize += copy; 8991da177e4SLinus Torvalds sk->sk_wmem_queued += copy; 9003ab224beSHideo Aoki sk_mem_charge(sk, copy); 90184fa7933SPatrick McHardy skb->ip_summed = CHECKSUM_PARTIAL; 9021da177e4SLinus Torvalds tp->write_seq += copy; 9031da177e4SLinus Torvalds TCP_SKB_CB(skb)->end_seq += copy; 9047967168cSHerbert Xu skb_shinfo(skb)->gso_segs = 0; 9051da177e4SLinus Torvalds 9061da177e4SLinus Torvalds if (!copied) 9074de075e0SEric Dumazet TCP_SKB_CB(skb)->tcp_flags &= ~TCPHDR_PSH; 9081da177e4SLinus Torvalds 9091da177e4SLinus Torvalds copied += copy; 91064022d0bSEric Dumazet offset += copy; 91164022d0bSEric Dumazet if (!(size -= copy)) 9121da177e4SLinus Torvalds goto out; 9131da177e4SLinus Torvalds 91469d15067SHerbert Xu if (skb->len < size_goal || (flags & MSG_OOB)) 9151da177e4SLinus Torvalds continue; 9161da177e4SLinus Torvalds 9171da177e4SLinus Torvalds if (forced_push(tp)) { 9181da177e4SLinus Torvalds tcp_mark_push(tp, skb); 9199e412ba7SIlpo Järvinen __tcp_push_pending_frames(sk, mss_now, TCP_NAGLE_PUSH); 920fe067e8aSDavid S. Miller } else if (skb == tcp_send_head(sk)) 9211da177e4SLinus Torvalds tcp_push_one(sk, mss_now); 9221da177e4SLinus Torvalds continue; 9231da177e4SLinus Torvalds 9241da177e4SLinus Torvalds wait_for_sndbuf: 9251da177e4SLinus Torvalds set_bit(SOCK_NOSPACE, &sk->sk_socket->flags); 9261da177e4SLinus Torvalds wait_for_memory: 9279e412ba7SIlpo Järvinen tcp_push(sk, flags & ~MSG_MORE, mss_now, TCP_NAGLE_PUSH); 9281da177e4SLinus Torvalds 9291da177e4SLinus Torvalds if ((err = sk_stream_wait_memory(sk, &timeo)) != 0) 9301da177e4SLinus Torvalds goto do_error; 9311da177e4SLinus Torvalds 9320c54b85fSIlpo Järvinen mss_now = tcp_send_mss(sk, &size_goal, flags); 9331da177e4SLinus Torvalds } 9341da177e4SLinus Torvalds 9351da177e4SLinus Torvalds out: 93635f9c09fSEric Dumazet if (copied && !(flags & MSG_SENDPAGE_NOTLAST)) 9379e412ba7SIlpo Järvinen tcp_push(sk, flags, mss_now, tp->nonagle); 9381da177e4SLinus Torvalds return copied; 9391da177e4SLinus Torvalds 9401da177e4SLinus Torvalds do_error: 9411da177e4SLinus Torvalds if (copied) 9421da177e4SLinus Torvalds goto out; 9431da177e4SLinus Torvalds out_err: 9441da177e4SLinus Torvalds return sk_stream_error(sk, flags, err); 9451da177e4SLinus Torvalds } 9461da177e4SLinus Torvalds 9477ba42910SChangli Gao int tcp_sendpage(struct sock *sk, struct page *page, int offset, 9481da177e4SLinus Torvalds size_t size, int flags) 9491da177e4SLinus Torvalds { 9501da177e4SLinus Torvalds ssize_t res; 9511da177e4SLinus Torvalds 9521da177e4SLinus Torvalds if (!(sk->sk_route_caps & NETIF_F_SG) || 9538648b305SHerbert Xu !(sk->sk_route_caps & NETIF_F_ALL_CSUM)) 9547ba42910SChangli Gao return sock_no_sendpage(sk->sk_socket, page, offset, size, 9557ba42910SChangli Gao flags); 9561da177e4SLinus Torvalds 9571da177e4SLinus Torvalds lock_sock(sk); 95864022d0bSEric Dumazet res = do_tcp_sendpages(sk, page, offset, size, flags); 9591da177e4SLinus Torvalds release_sock(sk); 9601da177e4SLinus Torvalds return res; 9611da177e4SLinus Torvalds } 9624bc2f18bSEric Dumazet EXPORT_SYMBOL(tcp_sendpage); 9631da177e4SLinus Torvalds 964690e99c4SEric Dumazet static inline int select_size(const struct sock *sk, bool sg) 9651da177e4SLinus Torvalds { 966cf533ea5SEric Dumazet const struct tcp_sock *tp = tcp_sk(sk); 967c1b4a7e6SDavid S. Miller int tmp = tp->mss_cache; 9681da177e4SLinus Torvalds 969def87cf4SKrishna Kumar if (sg) { 970f07d960dSEric Dumazet if (sk_can_gso(sk)) { 971f07d960dSEric Dumazet /* Small frames wont use a full page: 972f07d960dSEric Dumazet * Payload will immediately follow tcp header. 973f07d960dSEric Dumazet */ 974f07d960dSEric Dumazet tmp = SKB_WITH_OVERHEAD(2048 - MAX_TCP_HEADER); 975f07d960dSEric Dumazet } else { 976b4e26f5eSDavid S. Miller int pgbreak = SKB_MAX_HEAD(MAX_TCP_HEADER); 977b4e26f5eSDavid S. Miller 978b4e26f5eSDavid S. Miller if (tmp >= pgbreak && 979b4e26f5eSDavid S. Miller tmp <= pgbreak + (MAX_SKB_FRAGS - 1) * PAGE_SIZE) 980b4e26f5eSDavid S. Miller tmp = pgbreak; 981b4e26f5eSDavid S. Miller } 982b4e26f5eSDavid S. Miller } 9831da177e4SLinus Torvalds 9841da177e4SLinus Torvalds return tmp; 9851da177e4SLinus Torvalds } 9861da177e4SLinus Torvalds 987cf60af03SYuchung Cheng void tcp_free_fastopen_req(struct tcp_sock *tp) 988cf60af03SYuchung Cheng { 989cf60af03SYuchung Cheng if (tp->fastopen_req != NULL) { 990cf60af03SYuchung Cheng kfree(tp->fastopen_req); 991cf60af03SYuchung Cheng tp->fastopen_req = NULL; 992cf60af03SYuchung Cheng } 993cf60af03SYuchung Cheng } 994cf60af03SYuchung Cheng 995cf60af03SYuchung Cheng static int tcp_sendmsg_fastopen(struct sock *sk, struct msghdr *msg, int *size) 996cf60af03SYuchung Cheng { 997cf60af03SYuchung Cheng struct tcp_sock *tp = tcp_sk(sk); 998cf60af03SYuchung Cheng int err, flags; 999cf60af03SYuchung Cheng 1000cf60af03SYuchung Cheng if (!(sysctl_tcp_fastopen & TFO_CLIENT_ENABLE)) 1001cf60af03SYuchung Cheng return -EOPNOTSUPP; 1002cf60af03SYuchung Cheng if (tp->fastopen_req != NULL) 1003cf60af03SYuchung Cheng return -EALREADY; /* Another Fast Open is in progress */ 1004cf60af03SYuchung Cheng 1005cf60af03SYuchung Cheng tp->fastopen_req = kzalloc(sizeof(struct tcp_fastopen_request), 1006cf60af03SYuchung Cheng sk->sk_allocation); 1007cf60af03SYuchung Cheng if (unlikely(tp->fastopen_req == NULL)) 1008cf60af03SYuchung Cheng return -ENOBUFS; 1009cf60af03SYuchung Cheng tp->fastopen_req->data = msg; 1010cf60af03SYuchung Cheng 1011cf60af03SYuchung Cheng flags = (msg->msg_flags & MSG_DONTWAIT) ? O_NONBLOCK : 0; 1012cf60af03SYuchung Cheng err = __inet_stream_connect(sk->sk_socket, msg->msg_name, 1013cf60af03SYuchung Cheng msg->msg_namelen, flags); 1014cf60af03SYuchung Cheng *size = tp->fastopen_req->copied; 1015cf60af03SYuchung Cheng tcp_free_fastopen_req(tp); 1016cf60af03SYuchung Cheng return err; 1017cf60af03SYuchung Cheng } 1018cf60af03SYuchung Cheng 10197ba42910SChangli Gao int tcp_sendmsg(struct kiocb *iocb, struct sock *sk, struct msghdr *msg, 10201da177e4SLinus Torvalds size_t size) 10211da177e4SLinus Torvalds { 10221da177e4SLinus Torvalds struct iovec *iov; 10231da177e4SLinus Torvalds struct tcp_sock *tp = tcp_sk(sk); 10241da177e4SLinus Torvalds struct sk_buff *skb; 1025cf60af03SYuchung Cheng int iovlen, flags, err, copied = 0; 1026cf60af03SYuchung Cheng int mss_now = 0, size_goal, copied_syn = 0, offset = 0; 1027690e99c4SEric Dumazet bool sg; 10281da177e4SLinus Torvalds long timeo; 10291da177e4SLinus Torvalds 10301da177e4SLinus Torvalds lock_sock(sk); 10311da177e4SLinus Torvalds 10321da177e4SLinus Torvalds flags = msg->msg_flags; 1033cf60af03SYuchung Cheng if (flags & MSG_FASTOPEN) { 1034cf60af03SYuchung Cheng err = tcp_sendmsg_fastopen(sk, msg, &copied_syn); 1035cf60af03SYuchung Cheng if (err == -EINPROGRESS && copied_syn > 0) 1036cf60af03SYuchung Cheng goto out; 1037cf60af03SYuchung Cheng else if (err) 1038cf60af03SYuchung Cheng goto out_err; 1039cf60af03SYuchung Cheng offset = copied_syn; 1040cf60af03SYuchung Cheng } 1041cf60af03SYuchung Cheng 10421da177e4SLinus Torvalds timeo = sock_sndtimeo(sk, flags & MSG_DONTWAIT); 10431da177e4SLinus Torvalds 10448336886fSJerry Chu /* Wait for a connection to finish. One exception is TCP Fast Open 10458336886fSJerry Chu * (passive side) where data is allowed to be sent before a connection 10468336886fSJerry Chu * is fully established. 10478336886fSJerry Chu */ 10488336886fSJerry Chu if (((1 << sk->sk_state) & ~(TCPF_ESTABLISHED | TCPF_CLOSE_WAIT)) && 10498336886fSJerry Chu !tcp_passive_fastopen(sk)) { 10501da177e4SLinus Torvalds if ((err = sk_stream_wait_connect(sk, &timeo)) != 0) 1051cf60af03SYuchung Cheng goto do_error; 10528336886fSJerry Chu } 10531da177e4SLinus Torvalds 1054c0e88ff0SPavel Emelyanov if (unlikely(tp->repair)) { 1055c0e88ff0SPavel Emelyanov if (tp->repair_queue == TCP_RECV_QUEUE) { 1056c0e88ff0SPavel Emelyanov copied = tcp_send_rcvq(sk, msg, size); 1057c0e88ff0SPavel Emelyanov goto out; 1058c0e88ff0SPavel Emelyanov } 1059c0e88ff0SPavel Emelyanov 1060c0e88ff0SPavel Emelyanov err = -EINVAL; 1061c0e88ff0SPavel Emelyanov if (tp->repair_queue == TCP_NO_QUEUE) 1062c0e88ff0SPavel Emelyanov goto out_err; 1063c0e88ff0SPavel Emelyanov 1064c0e88ff0SPavel Emelyanov /* 'common' sending to sendq */ 1065c0e88ff0SPavel Emelyanov } 1066c0e88ff0SPavel Emelyanov 10671da177e4SLinus Torvalds /* This should be in poll */ 10681da177e4SLinus Torvalds clear_bit(SOCK_ASYNC_NOSPACE, &sk->sk_socket->flags); 10691da177e4SLinus Torvalds 10700c54b85fSIlpo Järvinen mss_now = tcp_send_mss(sk, &size_goal, flags); 10711da177e4SLinus Torvalds 10721da177e4SLinus Torvalds /* Ok commence sending. */ 10731da177e4SLinus Torvalds iovlen = msg->msg_iovlen; 10741da177e4SLinus Torvalds iov = msg->msg_iov; 10751da177e4SLinus Torvalds copied = 0; 10761da177e4SLinus Torvalds 10771da177e4SLinus Torvalds err = -EPIPE; 10781da177e4SLinus Torvalds if (sk->sk_err || (sk->sk_shutdown & SEND_SHUTDOWN)) 10790d6a775eSIlpo Järvinen goto out_err; 10801da177e4SLinus Torvalds 1081690e99c4SEric Dumazet sg = !!(sk->sk_route_caps & NETIF_F_SG); 1082def87cf4SKrishna Kumar 10831da177e4SLinus Torvalds while (--iovlen >= 0) { 108401db403cSDavid S. Miller size_t seglen = iov->iov_len; 10851da177e4SLinus Torvalds unsigned char __user *from = iov->iov_base; 10861da177e4SLinus Torvalds 10871da177e4SLinus Torvalds iov++; 1088cf60af03SYuchung Cheng if (unlikely(offset > 0)) { /* Skip bytes copied in SYN */ 1089cf60af03SYuchung Cheng if (offset >= seglen) { 1090cf60af03SYuchung Cheng offset -= seglen; 1091cf60af03SYuchung Cheng continue; 1092cf60af03SYuchung Cheng } 1093cf60af03SYuchung Cheng seglen -= offset; 1094cf60af03SYuchung Cheng from += offset; 1095cf60af03SYuchung Cheng offset = 0; 1096cf60af03SYuchung Cheng } 10971da177e4SLinus Torvalds 10981da177e4SLinus Torvalds while (seglen > 0) { 10996828b92bSHerbert Xu int copy = 0; 11006828b92bSHerbert Xu int max = size_goal; 11011da177e4SLinus Torvalds 1102fe067e8aSDavid S. Miller skb = tcp_write_queue_tail(sk); 11036828b92bSHerbert Xu if (tcp_send_head(sk)) { 11046828b92bSHerbert Xu if (skb->ip_summed == CHECKSUM_NONE) 11056828b92bSHerbert Xu max = mss_now; 11066828b92bSHerbert Xu copy = max - skb->len; 11076828b92bSHerbert Xu } 11081da177e4SLinus Torvalds 11096828b92bSHerbert Xu if (copy <= 0) { 11101da177e4SLinus Torvalds new_segment: 11111da177e4SLinus Torvalds /* Allocate new segment. If the interface is SG, 11121da177e4SLinus Torvalds * allocate skb fitting to single page. 11131da177e4SLinus Torvalds */ 11141da177e4SLinus Torvalds if (!sk_stream_memory_free(sk)) 11151da177e4SLinus Torvalds goto wait_for_sndbuf; 11161da177e4SLinus Torvalds 1117def87cf4SKrishna Kumar skb = sk_stream_alloc_skb(sk, 1118def87cf4SKrishna Kumar select_size(sk, sg), 1119df97c708SPavel Emelyanov sk->sk_allocation); 11201da177e4SLinus Torvalds if (!skb) 11211da177e4SLinus Torvalds goto wait_for_memory; 11221da177e4SLinus Torvalds 11231da177e4SLinus Torvalds /* 11241da177e4SLinus Torvalds * Check whether we can use HW checksum. 11251da177e4SLinus Torvalds */ 11268648b305SHerbert Xu if (sk->sk_route_caps & NETIF_F_ALL_CSUM) 112784fa7933SPatrick McHardy skb->ip_summed = CHECKSUM_PARTIAL; 11281da177e4SLinus Torvalds 11299e412ba7SIlpo Järvinen skb_entail(sk, skb); 1130c1b4a7e6SDavid S. Miller copy = size_goal; 11316828b92bSHerbert Xu max = size_goal; 11321da177e4SLinus Torvalds } 11331da177e4SLinus Torvalds 11341da177e4SLinus Torvalds /* Try to append data to the end of skb. */ 11351da177e4SLinus Torvalds if (copy > seglen) 11361da177e4SLinus Torvalds copy = seglen; 11371da177e4SLinus Torvalds 11381da177e4SLinus Torvalds /* Where to copy to? */ 1139a21d4572SEric Dumazet if (skb_availroom(skb) > 0) { 11401da177e4SLinus Torvalds /* We have some space in skb head. Superb! */ 1141a21d4572SEric Dumazet copy = min_t(int, copy, skb_availroom(skb)); 1142c6e1a0d1STom Herbert err = skb_add_data_nocache(sk, skb, from, copy); 1143c6e1a0d1STom Herbert if (err) 11441da177e4SLinus Torvalds goto do_fault; 11451da177e4SLinus Torvalds } else { 11465640f768SEric Dumazet bool merge = true; 11471da177e4SLinus Torvalds int i = skb_shinfo(skb)->nr_frags; 11485640f768SEric Dumazet struct page_frag *pfrag = sk_page_frag(sk); 1149761965eaSEric Dumazet 11505640f768SEric Dumazet if (!sk_page_frag_refill(sk, pfrag)) 11515640f768SEric Dumazet goto wait_for_memory; 1152761965eaSEric Dumazet 11535640f768SEric Dumazet if (!skb_can_coalesce(skb, i, pfrag->page, 11545640f768SEric Dumazet pfrag->offset)) { 11555640f768SEric Dumazet if (i == MAX_SKB_FRAGS || !sg) { 11561da177e4SLinus Torvalds tcp_mark_push(tp, skb); 11571da177e4SLinus Torvalds goto new_segment; 11581da177e4SLinus Torvalds } 11595640f768SEric Dumazet merge = false; 11605640f768SEric Dumazet } 1161ef015786SHerbert Xu 11625640f768SEric Dumazet copy = min_t(int, copy, pfrag->size - pfrag->offset); 1163ef015786SHerbert Xu 11643ab224beSHideo Aoki if (!sk_wmem_schedule(sk, copy)) 1165ef015786SHerbert Xu goto wait_for_memory; 11661da177e4SLinus Torvalds 1167c6e1a0d1STom Herbert err = skb_copy_to_page_nocache(sk, from, skb, 11685640f768SEric Dumazet pfrag->page, 11695640f768SEric Dumazet pfrag->offset, 11705640f768SEric Dumazet copy); 11715640f768SEric Dumazet if (err) 11721da177e4SLinus Torvalds goto do_error; 11731da177e4SLinus Torvalds 11741da177e4SLinus Torvalds /* Update the skb. */ 11751da177e4SLinus Torvalds if (merge) { 11769e903e08SEric Dumazet skb_frag_size_add(&skb_shinfo(skb)->frags[i - 1], copy); 11771da177e4SLinus Torvalds } else { 11785640f768SEric Dumazet skb_fill_page_desc(skb, i, pfrag->page, 11795640f768SEric Dumazet pfrag->offset, copy); 11805640f768SEric Dumazet get_page(pfrag->page); 11811da177e4SLinus Torvalds } 11825640f768SEric Dumazet pfrag->offset += copy; 11831da177e4SLinus Torvalds } 11841da177e4SLinus Torvalds 11851da177e4SLinus Torvalds if (!copied) 11864de075e0SEric Dumazet TCP_SKB_CB(skb)->tcp_flags &= ~TCPHDR_PSH; 11871da177e4SLinus Torvalds 11881da177e4SLinus Torvalds tp->write_seq += copy; 11891da177e4SLinus Torvalds TCP_SKB_CB(skb)->end_seq += copy; 11907967168cSHerbert Xu skb_shinfo(skb)->gso_segs = 0; 11911da177e4SLinus Torvalds 11921da177e4SLinus Torvalds from += copy; 11931da177e4SLinus Torvalds copied += copy; 11941da177e4SLinus Torvalds if ((seglen -= copy) == 0 && iovlen == 0) 11951da177e4SLinus Torvalds goto out; 11961da177e4SLinus Torvalds 1197c0e88ff0SPavel Emelyanov if (skb->len < max || (flags & MSG_OOB) || unlikely(tp->repair)) 11981da177e4SLinus Torvalds continue; 11991da177e4SLinus Torvalds 12001da177e4SLinus Torvalds if (forced_push(tp)) { 12011da177e4SLinus Torvalds tcp_mark_push(tp, skb); 12029e412ba7SIlpo Järvinen __tcp_push_pending_frames(sk, mss_now, TCP_NAGLE_PUSH); 1203fe067e8aSDavid S. Miller } else if (skb == tcp_send_head(sk)) 12041da177e4SLinus Torvalds tcp_push_one(sk, mss_now); 12051da177e4SLinus Torvalds continue; 12061da177e4SLinus Torvalds 12071da177e4SLinus Torvalds wait_for_sndbuf: 12081da177e4SLinus Torvalds set_bit(SOCK_NOSPACE, &sk->sk_socket->flags); 12091da177e4SLinus Torvalds wait_for_memory: 1210ec342325SAndrew Vagin if (copied) 12119e412ba7SIlpo Järvinen tcp_push(sk, flags & ~MSG_MORE, mss_now, TCP_NAGLE_PUSH); 12121da177e4SLinus Torvalds 12131da177e4SLinus Torvalds if ((err = sk_stream_wait_memory(sk, &timeo)) != 0) 12141da177e4SLinus Torvalds goto do_error; 12151da177e4SLinus Torvalds 12160c54b85fSIlpo Järvinen mss_now = tcp_send_mss(sk, &size_goal, flags); 12171da177e4SLinus Torvalds } 12181da177e4SLinus Torvalds } 12191da177e4SLinus Torvalds 12201da177e4SLinus Torvalds out: 1221ec342325SAndrew Vagin if (copied) 12229e412ba7SIlpo Järvinen tcp_push(sk, flags, mss_now, tp->nonagle); 12231da177e4SLinus Torvalds release_sock(sk); 1224cf60af03SYuchung Cheng return copied + copied_syn; 12251da177e4SLinus Torvalds 12261da177e4SLinus Torvalds do_fault: 12271da177e4SLinus Torvalds if (!skb->len) { 1228fe067e8aSDavid S. Miller tcp_unlink_write_queue(skb, sk); 1229fe067e8aSDavid S. Miller /* It is the one place in all of TCP, except connection 1230fe067e8aSDavid S. Miller * reset, where we can be unlinking the send_head. 1231fe067e8aSDavid S. Miller */ 1232fe067e8aSDavid S. Miller tcp_check_send_head(sk, skb); 12333ab224beSHideo Aoki sk_wmem_free_skb(sk, skb); 12341da177e4SLinus Torvalds } 12351da177e4SLinus Torvalds 12361da177e4SLinus Torvalds do_error: 1237cf60af03SYuchung Cheng if (copied + copied_syn) 12381da177e4SLinus Torvalds goto out; 12391da177e4SLinus Torvalds out_err: 12401da177e4SLinus Torvalds err = sk_stream_error(sk, flags, err); 12411da177e4SLinus Torvalds release_sock(sk); 12421da177e4SLinus Torvalds return err; 12431da177e4SLinus Torvalds } 12444bc2f18bSEric Dumazet EXPORT_SYMBOL(tcp_sendmsg); 12451da177e4SLinus Torvalds 12461da177e4SLinus Torvalds /* 12471da177e4SLinus Torvalds * Handle reading urgent data. BSD has very simple semantics for 12481da177e4SLinus Torvalds * this, no blocking and very strange errors 8) 12491da177e4SLinus Torvalds */ 12501da177e4SLinus Torvalds 1251377f0a08SRami Rosen static int tcp_recv_urg(struct sock *sk, struct msghdr *msg, int len, int flags) 12521da177e4SLinus Torvalds { 12531da177e4SLinus Torvalds struct tcp_sock *tp = tcp_sk(sk); 12541da177e4SLinus Torvalds 12551da177e4SLinus Torvalds /* No URG data to read. */ 12561da177e4SLinus Torvalds if (sock_flag(sk, SOCK_URGINLINE) || !tp->urg_data || 12571da177e4SLinus Torvalds tp->urg_data == TCP_URG_READ) 12581da177e4SLinus Torvalds return -EINVAL; /* Yes this is right ! */ 12591da177e4SLinus Torvalds 12601da177e4SLinus Torvalds if (sk->sk_state == TCP_CLOSE && !sock_flag(sk, SOCK_DONE)) 12611da177e4SLinus Torvalds return -ENOTCONN; 12621da177e4SLinus Torvalds 12631da177e4SLinus Torvalds if (tp->urg_data & TCP_URG_VALID) { 12641da177e4SLinus Torvalds int err = 0; 12651da177e4SLinus Torvalds char c = tp->urg_data; 12661da177e4SLinus Torvalds 12671da177e4SLinus Torvalds if (!(flags & MSG_PEEK)) 12681da177e4SLinus Torvalds tp->urg_data = TCP_URG_READ; 12691da177e4SLinus Torvalds 12701da177e4SLinus Torvalds /* Read urgent data. */ 12711da177e4SLinus Torvalds msg->msg_flags |= MSG_OOB; 12721da177e4SLinus Torvalds 12731da177e4SLinus Torvalds if (len > 0) { 12741da177e4SLinus Torvalds if (!(flags & MSG_TRUNC)) 12751da177e4SLinus Torvalds err = memcpy_toiovec(msg->msg_iov, &c, 1); 12761da177e4SLinus Torvalds len = 1; 12771da177e4SLinus Torvalds } else 12781da177e4SLinus Torvalds msg->msg_flags |= MSG_TRUNC; 12791da177e4SLinus Torvalds 12801da177e4SLinus Torvalds return err ? -EFAULT : len; 12811da177e4SLinus Torvalds } 12821da177e4SLinus Torvalds 12831da177e4SLinus Torvalds if (sk->sk_state == TCP_CLOSE || (sk->sk_shutdown & RCV_SHUTDOWN)) 12841da177e4SLinus Torvalds return 0; 12851da177e4SLinus Torvalds 12861da177e4SLinus Torvalds /* Fixed the recv(..., MSG_OOB) behaviour. BSD docs and 12871da177e4SLinus Torvalds * the available implementations agree in this case: 12881da177e4SLinus Torvalds * this call should never block, independent of the 12891da177e4SLinus Torvalds * blocking state of the socket. 12901da177e4SLinus Torvalds * Mike <pall@rz.uni-karlsruhe.de> 12911da177e4SLinus Torvalds */ 12921da177e4SLinus Torvalds return -EAGAIN; 12931da177e4SLinus Torvalds } 12941da177e4SLinus Torvalds 1295c0e88ff0SPavel Emelyanov static int tcp_peek_sndq(struct sock *sk, struct msghdr *msg, int len) 1296c0e88ff0SPavel Emelyanov { 1297c0e88ff0SPavel Emelyanov struct sk_buff *skb; 1298c0e88ff0SPavel Emelyanov int copied = 0, err = 0; 1299c0e88ff0SPavel Emelyanov 1300c0e88ff0SPavel Emelyanov /* XXX -- need to support SO_PEEK_OFF */ 1301c0e88ff0SPavel Emelyanov 1302c0e88ff0SPavel Emelyanov skb_queue_walk(&sk->sk_write_queue, skb) { 1303c0e88ff0SPavel Emelyanov err = skb_copy_datagram_iovec(skb, 0, msg->msg_iov, skb->len); 1304c0e88ff0SPavel Emelyanov if (err) 1305c0e88ff0SPavel Emelyanov break; 1306c0e88ff0SPavel Emelyanov 1307c0e88ff0SPavel Emelyanov copied += skb->len; 1308c0e88ff0SPavel Emelyanov } 1309c0e88ff0SPavel Emelyanov 1310c0e88ff0SPavel Emelyanov return err ?: copied; 1311c0e88ff0SPavel Emelyanov } 1312c0e88ff0SPavel Emelyanov 13131da177e4SLinus Torvalds /* Clean up the receive buffer for full frames taken by the user, 13141da177e4SLinus Torvalds * then send an ACK if necessary. COPIED is the number of bytes 13151da177e4SLinus Torvalds * tcp_recvmsg has given to the user so far, it speeds up the 13161da177e4SLinus Torvalds * calculation of whether or not we must ACK for the sake of 13171da177e4SLinus Torvalds * a window update. 13181da177e4SLinus Torvalds */ 13190e4b4992SChris Leech void tcp_cleanup_rbuf(struct sock *sk, int copied) 13201da177e4SLinus Torvalds { 13211da177e4SLinus Torvalds struct tcp_sock *tp = tcp_sk(sk); 1322a2a385d6SEric Dumazet bool time_to_ack = false; 13231da177e4SLinus Torvalds 13241da177e4SLinus Torvalds struct sk_buff *skb = skb_peek(&sk->sk_receive_queue); 13251da177e4SLinus Torvalds 1326d792c100SIlpo Järvinen WARN(skb && !before(tp->copied_seq, TCP_SKB_CB(skb)->end_seq), 13272af6fd8bSJoe Perches "cleanup rbuf bug: copied %X seq %X rcvnxt %X\n", 1328d792c100SIlpo Järvinen tp->copied_seq, TCP_SKB_CB(skb)->end_seq, tp->rcv_nxt); 13291da177e4SLinus Torvalds 1330463c84b9SArnaldo Carvalho de Melo if (inet_csk_ack_scheduled(sk)) { 1331463c84b9SArnaldo Carvalho de Melo const struct inet_connection_sock *icsk = inet_csk(sk); 13321da177e4SLinus Torvalds /* Delayed ACKs frequently hit locked sockets during bulk 13331da177e4SLinus Torvalds * receive. */ 1334463c84b9SArnaldo Carvalho de Melo if (icsk->icsk_ack.blocked || 13351da177e4SLinus Torvalds /* Once-per-two-segments ACK was not sent by tcp_input.c */ 1336463c84b9SArnaldo Carvalho de Melo tp->rcv_nxt - tp->rcv_wup > icsk->icsk_ack.rcv_mss || 13371da177e4SLinus Torvalds /* 13381da177e4SLinus Torvalds * If this read emptied read buffer, we send ACK, if 13391da177e4SLinus Torvalds * connection is not bidirectional, user drained 13401da177e4SLinus Torvalds * receive buffer and there was a small segment 13411da177e4SLinus Torvalds * in queue. 13421da177e4SLinus Torvalds */ 13431ef9696cSAlexey Kuznetsov (copied > 0 && 13441ef9696cSAlexey Kuznetsov ((icsk->icsk_ack.pending & ICSK_ACK_PUSHED2) || 13451ef9696cSAlexey Kuznetsov ((icsk->icsk_ack.pending & ICSK_ACK_PUSHED) && 13461ef9696cSAlexey Kuznetsov !icsk->icsk_ack.pingpong)) && 13471ef9696cSAlexey Kuznetsov !atomic_read(&sk->sk_rmem_alloc))) 1348a2a385d6SEric Dumazet time_to_ack = true; 13491da177e4SLinus Torvalds } 13501da177e4SLinus Torvalds 13511da177e4SLinus Torvalds /* We send an ACK if we can now advertise a non-zero window 13521da177e4SLinus Torvalds * which has been raised "significantly". 13531da177e4SLinus Torvalds * 13541da177e4SLinus Torvalds * Even if window raised up to infinity, do not send window open ACK 13551da177e4SLinus Torvalds * in states, where we will not receive more. It is useless. 13561da177e4SLinus Torvalds */ 13571da177e4SLinus Torvalds if (copied > 0 && !time_to_ack && !(sk->sk_shutdown & RCV_SHUTDOWN)) { 13581da177e4SLinus Torvalds __u32 rcv_window_now = tcp_receive_window(tp); 13591da177e4SLinus Torvalds 13601da177e4SLinus Torvalds /* Optimize, __tcp_select_window() is not cheap. */ 13611da177e4SLinus Torvalds if (2*rcv_window_now <= tp->window_clamp) { 13621da177e4SLinus Torvalds __u32 new_window = __tcp_select_window(sk); 13631da177e4SLinus Torvalds 13641da177e4SLinus Torvalds /* Send ACK now, if this read freed lots of space 13651da177e4SLinus Torvalds * in our buffer. Certainly, new_window is new window. 13661da177e4SLinus Torvalds * We can advertise it now, if it is not less than current one. 13671da177e4SLinus Torvalds * "Lots" means "at least twice" here. 13681da177e4SLinus Torvalds */ 13691da177e4SLinus Torvalds if (new_window && new_window >= 2 * rcv_window_now) 1370a2a385d6SEric Dumazet time_to_ack = true; 13711da177e4SLinus Torvalds } 13721da177e4SLinus Torvalds } 13731da177e4SLinus Torvalds if (time_to_ack) 13741da177e4SLinus Torvalds tcp_send_ack(sk); 13751da177e4SLinus Torvalds } 13761da177e4SLinus Torvalds 13771da177e4SLinus Torvalds static void tcp_prequeue_process(struct sock *sk) 13781da177e4SLinus Torvalds { 13791da177e4SLinus Torvalds struct sk_buff *skb; 13801da177e4SLinus Torvalds struct tcp_sock *tp = tcp_sk(sk); 13811da177e4SLinus Torvalds 13826f67c817SPavel Emelyanov NET_INC_STATS_USER(sock_net(sk), LINUX_MIB_TCPPREQUEUED); 13831da177e4SLinus Torvalds 13841da177e4SLinus Torvalds /* RX process wants to run with disabled BHs, though it is not 13851da177e4SLinus Torvalds * necessary */ 13861da177e4SLinus Torvalds local_bh_disable(); 13871da177e4SLinus Torvalds while ((skb = __skb_dequeue(&tp->ucopy.prequeue)) != NULL) 1388c57943a1SPeter Zijlstra sk_backlog_rcv(sk, skb); 13891da177e4SLinus Torvalds local_bh_enable(); 13901da177e4SLinus Torvalds 13911da177e4SLinus Torvalds /* Clear memory counter. */ 13921da177e4SLinus Torvalds tp->ucopy.memory = 0; 13931da177e4SLinus Torvalds } 13941da177e4SLinus Torvalds 139573852e81SSteven J. Magnani #ifdef CONFIG_NET_DMA 139673852e81SSteven J. Magnani static void tcp_service_net_dma(struct sock *sk, bool wait) 139773852e81SSteven J. Magnani { 139873852e81SSteven J. Magnani dma_cookie_t done, used; 139973852e81SSteven J. Magnani dma_cookie_t last_issued; 140073852e81SSteven J. Magnani struct tcp_sock *tp = tcp_sk(sk); 140173852e81SSteven J. Magnani 140273852e81SSteven J. Magnani if (!tp->ucopy.dma_chan) 140373852e81SSteven J. Magnani return; 140473852e81SSteven J. Magnani 140573852e81SSteven J. Magnani last_issued = tp->ucopy.dma_cookie; 1406b9ee8683SBartlomiej Zolnierkiewicz dma_async_issue_pending(tp->ucopy.dma_chan); 140773852e81SSteven J. Magnani 140873852e81SSteven J. Magnani do { 1409e239345fSBartlomiej Zolnierkiewicz if (dma_async_is_tx_complete(tp->ucopy.dma_chan, 141073852e81SSteven J. Magnani last_issued, &done, 141173852e81SSteven J. Magnani &used) == DMA_SUCCESS) { 141273852e81SSteven J. Magnani /* Safe to free early-copied skbs now */ 141373852e81SSteven J. Magnani __skb_queue_purge(&sk->sk_async_wait_queue); 141473852e81SSteven J. Magnani break; 141573852e81SSteven J. Magnani } else { 141673852e81SSteven J. Magnani struct sk_buff *skb; 141773852e81SSteven J. Magnani while ((skb = skb_peek(&sk->sk_async_wait_queue)) && 141873852e81SSteven J. Magnani (dma_async_is_complete(skb->dma_cookie, done, 141973852e81SSteven J. Magnani used) == DMA_SUCCESS)) { 142073852e81SSteven J. Magnani __skb_dequeue(&sk->sk_async_wait_queue); 142173852e81SSteven J. Magnani kfree_skb(skb); 142273852e81SSteven J. Magnani } 142373852e81SSteven J. Magnani } 142473852e81SSteven J. Magnani } while (wait); 142573852e81SSteven J. Magnani } 142673852e81SSteven J. Magnani #endif 142773852e81SSteven J. Magnani 1428f26845b4SEric Dumazet static struct sk_buff *tcp_recv_skb(struct sock *sk, u32 seq, u32 *off) 14291da177e4SLinus Torvalds { 14301da177e4SLinus Torvalds struct sk_buff *skb; 14311da177e4SLinus Torvalds u32 offset; 14321da177e4SLinus Torvalds 1433f26845b4SEric Dumazet while ((skb = skb_peek(&sk->sk_receive_queue)) != NULL) { 14341da177e4SLinus Torvalds offset = seq - TCP_SKB_CB(skb)->seq; 1435aa8223c7SArnaldo Carvalho de Melo if (tcp_hdr(skb)->syn) 14361da177e4SLinus Torvalds offset--; 1437aa8223c7SArnaldo Carvalho de Melo if (offset < skb->len || tcp_hdr(skb)->fin) { 14381da177e4SLinus Torvalds *off = offset; 14391da177e4SLinus Torvalds return skb; 14401da177e4SLinus Torvalds } 1441f26845b4SEric Dumazet /* This looks weird, but this can happen if TCP collapsing 1442f26845b4SEric Dumazet * splitted a fat GRO packet, while we released socket lock 1443f26845b4SEric Dumazet * in skb_splice_bits() 1444f26845b4SEric Dumazet */ 1445f26845b4SEric Dumazet sk_eat_skb(sk, skb, false); 14461da177e4SLinus Torvalds } 14471da177e4SLinus Torvalds return NULL; 14481da177e4SLinus Torvalds } 14491da177e4SLinus Torvalds 14501da177e4SLinus Torvalds /* 14511da177e4SLinus Torvalds * This routine provides an alternative to tcp_recvmsg() for routines 14521da177e4SLinus Torvalds * that would like to handle copying from skbuffs directly in 'sendfile' 14531da177e4SLinus Torvalds * fashion. 14541da177e4SLinus Torvalds * Note: 14551da177e4SLinus Torvalds * - It is assumed that the socket was locked by the caller. 14561da177e4SLinus Torvalds * - The routine does not block. 14571da177e4SLinus Torvalds * - At present, there is no support for reading OOB data 14581da177e4SLinus Torvalds * or for 'peeking' the socket using this routine 14591da177e4SLinus Torvalds * (although both would be easy to implement). 14601da177e4SLinus Torvalds */ 14611da177e4SLinus Torvalds int tcp_read_sock(struct sock *sk, read_descriptor_t *desc, 14621da177e4SLinus Torvalds sk_read_actor_t recv_actor) 14631da177e4SLinus Torvalds { 14641da177e4SLinus Torvalds struct sk_buff *skb; 14651da177e4SLinus Torvalds struct tcp_sock *tp = tcp_sk(sk); 14661da177e4SLinus Torvalds u32 seq = tp->copied_seq; 14671da177e4SLinus Torvalds u32 offset; 14681da177e4SLinus Torvalds int copied = 0; 14691da177e4SLinus Torvalds 14701da177e4SLinus Torvalds if (sk->sk_state == TCP_LISTEN) 14711da177e4SLinus Torvalds return -ENOTCONN; 14721da177e4SLinus Torvalds while ((skb = tcp_recv_skb(sk, seq, &offset)) != NULL) { 14731da177e4SLinus Torvalds if (offset < skb->len) { 1474374e7b59SOctavian Purdila int used; 1475374e7b59SOctavian Purdila size_t len; 14761da177e4SLinus Torvalds 14771da177e4SLinus Torvalds len = skb->len - offset; 14781da177e4SLinus Torvalds /* Stop reading if we hit a patch of urgent data */ 14791da177e4SLinus Torvalds if (tp->urg_data) { 14801da177e4SLinus Torvalds u32 urg_offset = tp->urg_seq - seq; 14811da177e4SLinus Torvalds if (urg_offset < len) 14821da177e4SLinus Torvalds len = urg_offset; 14831da177e4SLinus Torvalds if (!len) 14841da177e4SLinus Torvalds break; 14851da177e4SLinus Torvalds } 14861da177e4SLinus Torvalds used = recv_actor(desc, skb, offset, len); 1487ff905b1eSEric Dumazet if (used <= 0) { 1488ddb61a57SJens Axboe if (!copied) 1489ddb61a57SJens Axboe copied = used; 1490ddb61a57SJens Axboe break; 1491ddb61a57SJens Axboe } else if (used <= len) { 14921da177e4SLinus Torvalds seq += used; 14931da177e4SLinus Torvalds copied += used; 14941da177e4SLinus Torvalds offset += used; 14951da177e4SLinus Torvalds } 149602275a2eSWilly Tarreau /* If recv_actor drops the lock (e.g. TCP splice 1497293ad604SOctavian Purdila * receive) the skb pointer might be invalid when 1498293ad604SOctavian Purdila * getting here: tcp_collapse might have deleted it 1499293ad604SOctavian Purdila * while aggregating skbs from the socket queue. 1500293ad604SOctavian Purdila */ 1501293ad604SOctavian Purdila skb = tcp_recv_skb(sk, seq - 1, &offset); 150202275a2eSWilly Tarreau if (!skb) 15031da177e4SLinus Torvalds break; 150402275a2eSWilly Tarreau /* TCP coalescing might have appended data to the skb. 150502275a2eSWilly Tarreau * Try to splice more frags 150602275a2eSWilly Tarreau */ 150702275a2eSWilly Tarreau if (offset + 1 != skb->len) 150802275a2eSWilly Tarreau continue; 15091da177e4SLinus Torvalds } 1510aa8223c7SArnaldo Carvalho de Melo if (tcp_hdr(skb)->fin) { 1511dc6b9b78SEric Dumazet sk_eat_skb(sk, skb, false); 15121da177e4SLinus Torvalds ++seq; 15131da177e4SLinus Torvalds break; 15141da177e4SLinus Torvalds } 1515dc6b9b78SEric Dumazet sk_eat_skb(sk, skb, false); 15161da177e4SLinus Torvalds if (!desc->count) 15171da177e4SLinus Torvalds break; 1518baff42abSSteven J. Magnani tp->copied_seq = seq; 15191da177e4SLinus Torvalds } 15201da177e4SLinus Torvalds tp->copied_seq = seq; 15211da177e4SLinus Torvalds 15221da177e4SLinus Torvalds tcp_rcv_space_adjust(sk); 15231da177e4SLinus Torvalds 15241da177e4SLinus Torvalds /* Clean up data we have read: This will do ACK frames. */ 1525f26845b4SEric Dumazet if (copied > 0) { 1526f26845b4SEric Dumazet tcp_recv_skb(sk, seq, &offset); 15270e4b4992SChris Leech tcp_cleanup_rbuf(sk, copied); 1528f26845b4SEric Dumazet } 15291da177e4SLinus Torvalds return copied; 15301da177e4SLinus Torvalds } 15314bc2f18bSEric Dumazet EXPORT_SYMBOL(tcp_read_sock); 15321da177e4SLinus Torvalds 15331da177e4SLinus Torvalds /* 15341da177e4SLinus Torvalds * This routine copies from a sock struct into the user buffer. 15351da177e4SLinus Torvalds * 15361da177e4SLinus Torvalds * Technical note: in 2.3 we work on _locked_ socket, so that 15371da177e4SLinus Torvalds * tricks with *seq access order and skb->users are not required. 15381da177e4SLinus Torvalds * Probably, code can be easily improved even more. 15391da177e4SLinus Torvalds */ 15401da177e4SLinus Torvalds 15411da177e4SLinus Torvalds int tcp_recvmsg(struct kiocb *iocb, struct sock *sk, struct msghdr *msg, 15421da177e4SLinus Torvalds size_t len, int nonblock, int flags, int *addr_len) 15431da177e4SLinus Torvalds { 15441da177e4SLinus Torvalds struct tcp_sock *tp = tcp_sk(sk); 15451da177e4SLinus Torvalds int copied = 0; 15461da177e4SLinus Torvalds u32 peek_seq; 15471da177e4SLinus Torvalds u32 *seq; 15481da177e4SLinus Torvalds unsigned long used; 15491da177e4SLinus Torvalds int err; 15501da177e4SLinus Torvalds int target; /* Read at least this many bytes */ 15511da177e4SLinus Torvalds long timeo; 15521da177e4SLinus Torvalds struct task_struct *user_recv = NULL; 1553dc6b9b78SEric Dumazet bool copied_early = false; 15542b1244a4SChris Leech struct sk_buff *skb; 155577527313SIlpo Järvinen u32 urg_hole = 0; 15561da177e4SLinus Torvalds 1557cbf55001SEliezer Tamir if (sk_can_busy_loop(sk) && skb_queue_empty(&sk->sk_receive_queue) && 1558cbf55001SEliezer Tamir (sk->sk_state == TCP_ESTABLISHED)) 1559cbf55001SEliezer Tamir sk_busy_loop(sk, nonblock); 1560d30e383bSEliezer Tamir 15611da177e4SLinus Torvalds lock_sock(sk); 15621da177e4SLinus Torvalds 15631da177e4SLinus Torvalds err = -ENOTCONN; 15641da177e4SLinus Torvalds if (sk->sk_state == TCP_LISTEN) 15651da177e4SLinus Torvalds goto out; 15661da177e4SLinus Torvalds 15671da177e4SLinus Torvalds timeo = sock_rcvtimeo(sk, nonblock); 15681da177e4SLinus Torvalds 15691da177e4SLinus Torvalds /* Urgent data needs to be handled specially. */ 15701da177e4SLinus Torvalds if (flags & MSG_OOB) 15711da177e4SLinus Torvalds goto recv_urg; 15721da177e4SLinus Torvalds 1573c0e88ff0SPavel Emelyanov if (unlikely(tp->repair)) { 1574c0e88ff0SPavel Emelyanov err = -EPERM; 1575c0e88ff0SPavel Emelyanov if (!(flags & MSG_PEEK)) 1576c0e88ff0SPavel Emelyanov goto out; 1577c0e88ff0SPavel Emelyanov 1578c0e88ff0SPavel Emelyanov if (tp->repair_queue == TCP_SEND_QUEUE) 1579c0e88ff0SPavel Emelyanov goto recv_sndq; 1580c0e88ff0SPavel Emelyanov 1581c0e88ff0SPavel Emelyanov err = -EINVAL; 1582c0e88ff0SPavel Emelyanov if (tp->repair_queue == TCP_NO_QUEUE) 1583c0e88ff0SPavel Emelyanov goto out; 1584c0e88ff0SPavel Emelyanov 1585c0e88ff0SPavel Emelyanov /* 'common' recv queue MSG_PEEK-ing */ 1586c0e88ff0SPavel Emelyanov } 1587c0e88ff0SPavel Emelyanov 15881da177e4SLinus Torvalds seq = &tp->copied_seq; 15891da177e4SLinus Torvalds if (flags & MSG_PEEK) { 15901da177e4SLinus Torvalds peek_seq = tp->copied_seq; 15911da177e4SLinus Torvalds seq = &peek_seq; 15921da177e4SLinus Torvalds } 15931da177e4SLinus Torvalds 15941da177e4SLinus Torvalds target = sock_rcvlowat(sk, flags & MSG_WAITALL, len); 15951da177e4SLinus Torvalds 15961a2449a8SChris Leech #ifdef CONFIG_NET_DMA 15971a2449a8SChris Leech tp->ucopy.dma_chan = NULL; 15981a2449a8SChris Leech preempt_disable(); 15992b1244a4SChris Leech skb = skb_peek_tail(&sk->sk_receive_queue); 1600e00c5d8bSAndrew Morton { 1601e00c5d8bSAndrew Morton int available = 0; 1602e00c5d8bSAndrew Morton 16032b1244a4SChris Leech if (skb) 16042b1244a4SChris Leech available = TCP_SKB_CB(skb)->seq + skb->len - (*seq); 16052b1244a4SChris Leech if ((available < target) && 16062b1244a4SChris Leech (len > sysctl_tcp_dma_copybreak) && !(flags & MSG_PEEK) && 1607e00c5d8bSAndrew Morton !sysctl_tcp_low_latency && 1608a2bd1140SDave Jiang net_dma_find_channel()) { 16091a2449a8SChris Leech preempt_enable_no_resched(); 1610e00c5d8bSAndrew Morton tp->ucopy.pinned_list = 1611e00c5d8bSAndrew Morton dma_pin_iovec_pages(msg->msg_iov, len); 1612e00c5d8bSAndrew Morton } else { 16131a2449a8SChris Leech preempt_enable_no_resched(); 1614e00c5d8bSAndrew Morton } 1615e00c5d8bSAndrew Morton } 16161a2449a8SChris Leech #endif 16171a2449a8SChris Leech 16181da177e4SLinus Torvalds do { 16191da177e4SLinus Torvalds u32 offset; 16201da177e4SLinus Torvalds 16211da177e4SLinus Torvalds /* Are we at urgent data? Stop if we have read anything or have SIGURG pending. */ 16221da177e4SLinus Torvalds if (tp->urg_data && tp->urg_seq == *seq) { 16231da177e4SLinus Torvalds if (copied) 16241da177e4SLinus Torvalds break; 16251da177e4SLinus Torvalds if (signal_pending(current)) { 16261da177e4SLinus Torvalds copied = timeo ? sock_intr_errno(timeo) : -EAGAIN; 16271da177e4SLinus Torvalds break; 16281da177e4SLinus Torvalds } 16291da177e4SLinus Torvalds } 16301da177e4SLinus Torvalds 16311da177e4SLinus Torvalds /* Next get a buffer. */ 16321da177e4SLinus Torvalds 163391521944SDavid S. Miller skb_queue_walk(&sk->sk_receive_queue, skb) { 16341da177e4SLinus Torvalds /* Now that we have two receive queues this 16351da177e4SLinus Torvalds * shouldn't happen. 16361da177e4SLinus Torvalds */ 1637d792c100SIlpo Järvinen if (WARN(before(*seq, TCP_SKB_CB(skb)->seq), 16382af6fd8bSJoe Perches "recvmsg bug: copied %X seq %X rcvnxt %X fl %X\n", 16392af6fd8bSJoe Perches *seq, TCP_SKB_CB(skb)->seq, tp->rcv_nxt, 1640d792c100SIlpo Järvinen flags)) 16411da177e4SLinus Torvalds break; 1642d792c100SIlpo Järvinen 16431da177e4SLinus Torvalds offset = *seq - TCP_SKB_CB(skb)->seq; 1644aa8223c7SArnaldo Carvalho de Melo if (tcp_hdr(skb)->syn) 16451da177e4SLinus Torvalds offset--; 16461da177e4SLinus Torvalds if (offset < skb->len) 16471da177e4SLinus Torvalds goto found_ok_skb; 1648aa8223c7SArnaldo Carvalho de Melo if (tcp_hdr(skb)->fin) 16491da177e4SLinus Torvalds goto found_fin_ok; 16502af6fd8bSJoe Perches WARN(!(flags & MSG_PEEK), 16512af6fd8bSJoe Perches "recvmsg bug 2: copied %X seq %X rcvnxt %X fl %X\n", 16522af6fd8bSJoe Perches *seq, TCP_SKB_CB(skb)->seq, tp->rcv_nxt, flags); 165391521944SDavid S. Miller } 16541da177e4SLinus Torvalds 16551da177e4SLinus Torvalds /* Well, if we have backlog, try to process it now yet. */ 16561da177e4SLinus Torvalds 16571da177e4SLinus Torvalds if (copied >= target && !sk->sk_backlog.tail) 16581da177e4SLinus Torvalds break; 16591da177e4SLinus Torvalds 16601da177e4SLinus Torvalds if (copied) { 16611da177e4SLinus Torvalds if (sk->sk_err || 16621da177e4SLinus Torvalds sk->sk_state == TCP_CLOSE || 16631da177e4SLinus Torvalds (sk->sk_shutdown & RCV_SHUTDOWN) || 16641da177e4SLinus Torvalds !timeo || 1665518a09efSDavid S. Miller signal_pending(current)) 16661da177e4SLinus Torvalds break; 16671da177e4SLinus Torvalds } else { 16681da177e4SLinus Torvalds if (sock_flag(sk, SOCK_DONE)) 16691da177e4SLinus Torvalds break; 16701da177e4SLinus Torvalds 16711da177e4SLinus Torvalds if (sk->sk_err) { 16721da177e4SLinus Torvalds copied = sock_error(sk); 16731da177e4SLinus Torvalds break; 16741da177e4SLinus Torvalds } 16751da177e4SLinus Torvalds 16761da177e4SLinus Torvalds if (sk->sk_shutdown & RCV_SHUTDOWN) 16771da177e4SLinus Torvalds break; 16781da177e4SLinus Torvalds 16791da177e4SLinus Torvalds if (sk->sk_state == TCP_CLOSE) { 16801da177e4SLinus Torvalds if (!sock_flag(sk, SOCK_DONE)) { 16811da177e4SLinus Torvalds /* This occurs when user tries to read 16821da177e4SLinus Torvalds * from never connected socket. 16831da177e4SLinus Torvalds */ 16841da177e4SLinus Torvalds copied = -ENOTCONN; 16851da177e4SLinus Torvalds break; 16861da177e4SLinus Torvalds } 16871da177e4SLinus Torvalds break; 16881da177e4SLinus Torvalds } 16891da177e4SLinus Torvalds 16901da177e4SLinus Torvalds if (!timeo) { 16911da177e4SLinus Torvalds copied = -EAGAIN; 16921da177e4SLinus Torvalds break; 16931da177e4SLinus Torvalds } 16941da177e4SLinus Torvalds 16951da177e4SLinus Torvalds if (signal_pending(current)) { 16961da177e4SLinus Torvalds copied = sock_intr_errno(timeo); 16971da177e4SLinus Torvalds break; 16981da177e4SLinus Torvalds } 16991da177e4SLinus Torvalds } 17001da177e4SLinus Torvalds 17010e4b4992SChris Leech tcp_cleanup_rbuf(sk, copied); 17021da177e4SLinus Torvalds 17037df55125SDavid S. Miller if (!sysctl_tcp_low_latency && tp->ucopy.task == user_recv) { 17041da177e4SLinus Torvalds /* Install new reader */ 17051da177e4SLinus Torvalds if (!user_recv && !(flags & (MSG_TRUNC | MSG_PEEK))) { 17061da177e4SLinus Torvalds user_recv = current; 17071da177e4SLinus Torvalds tp->ucopy.task = user_recv; 17081da177e4SLinus Torvalds tp->ucopy.iov = msg->msg_iov; 17091da177e4SLinus Torvalds } 17101da177e4SLinus Torvalds 17111da177e4SLinus Torvalds tp->ucopy.len = len; 17121da177e4SLinus Torvalds 1713547b792cSIlpo Järvinen WARN_ON(tp->copied_seq != tp->rcv_nxt && 1714547b792cSIlpo Järvinen !(flags & (MSG_PEEK | MSG_TRUNC))); 17151da177e4SLinus Torvalds 17161da177e4SLinus Torvalds /* Ugly... If prequeue is not empty, we have to 17171da177e4SLinus Torvalds * process it before releasing socket, otherwise 17181da177e4SLinus Torvalds * order will be broken at second iteration. 17191da177e4SLinus Torvalds * More elegant solution is required!!! 17201da177e4SLinus Torvalds * 17211da177e4SLinus Torvalds * Look: we have the following (pseudo)queues: 17221da177e4SLinus Torvalds * 17231da177e4SLinus Torvalds * 1. packets in flight 17241da177e4SLinus Torvalds * 2. backlog 17251da177e4SLinus Torvalds * 3. prequeue 17261da177e4SLinus Torvalds * 4. receive_queue 17271da177e4SLinus Torvalds * 17281da177e4SLinus Torvalds * Each queue can be processed only if the next ones 17291da177e4SLinus Torvalds * are empty. At this point we have empty receive_queue. 17301da177e4SLinus Torvalds * But prequeue _can_ be not empty after 2nd iteration, 17311da177e4SLinus Torvalds * when we jumped to start of loop because backlog 17321da177e4SLinus Torvalds * processing added something to receive_queue. 17331da177e4SLinus Torvalds * We cannot release_sock(), because backlog contains 17341da177e4SLinus Torvalds * packets arrived _after_ prequeued ones. 17351da177e4SLinus Torvalds * 17361da177e4SLinus Torvalds * Shortly, algorithm is clear --- to process all 17371da177e4SLinus Torvalds * the queues in order. We could make it more directly, 17381da177e4SLinus Torvalds * requeueing packets from backlog to prequeue, if 17391da177e4SLinus Torvalds * is not empty. It is more elegant, but eats cycles, 17401da177e4SLinus Torvalds * unfortunately. 17411da177e4SLinus Torvalds */ 1742b03efcfbSDavid S. Miller if (!skb_queue_empty(&tp->ucopy.prequeue)) 17431da177e4SLinus Torvalds goto do_prequeue; 17441da177e4SLinus Torvalds 17451da177e4SLinus Torvalds /* __ Set realtime policy in scheduler __ */ 17461da177e4SLinus Torvalds } 17471da177e4SLinus Torvalds 174873852e81SSteven J. Magnani #ifdef CONFIG_NET_DMA 174915c04175SMichal Kubeček if (tp->ucopy.dma_chan) { 175015c04175SMichal Kubeček if (tp->rcv_wnd == 0 && 175115c04175SMichal Kubeček !skb_queue_empty(&sk->sk_async_wait_queue)) { 175215c04175SMichal Kubeček tcp_service_net_dma(sk, true); 175315c04175SMichal Kubeček tcp_cleanup_rbuf(sk, copied); 175415c04175SMichal Kubeček } else 1755b9ee8683SBartlomiej Zolnierkiewicz dma_async_issue_pending(tp->ucopy.dma_chan); 175615c04175SMichal Kubeček } 175773852e81SSteven J. Magnani #endif 17581da177e4SLinus Torvalds if (copied >= target) { 17591da177e4SLinus Torvalds /* Do not sleep, just process backlog. */ 17601da177e4SLinus Torvalds release_sock(sk); 17611da177e4SLinus Torvalds lock_sock(sk); 17621da177e4SLinus Torvalds } else 17631da177e4SLinus Torvalds sk_wait_data(sk, &timeo); 17641da177e4SLinus Torvalds 17651a2449a8SChris Leech #ifdef CONFIG_NET_DMA 176673852e81SSteven J. Magnani tcp_service_net_dma(sk, false); /* Don't block */ 17671a2449a8SChris Leech tp->ucopy.wakeup = 0; 17681a2449a8SChris Leech #endif 17691a2449a8SChris Leech 17701da177e4SLinus Torvalds if (user_recv) { 17711da177e4SLinus Torvalds int chunk; 17721da177e4SLinus Torvalds 17731da177e4SLinus Torvalds /* __ Restore normal policy in scheduler __ */ 17741da177e4SLinus Torvalds 17751da177e4SLinus Torvalds if ((chunk = len - tp->ucopy.len) != 0) { 1776ed88098eSPavel Emelyanov NET_ADD_STATS_USER(sock_net(sk), LINUX_MIB_TCPDIRECTCOPYFROMBACKLOG, chunk); 17771da177e4SLinus Torvalds len -= chunk; 17781da177e4SLinus Torvalds copied += chunk; 17791da177e4SLinus Torvalds } 17801da177e4SLinus Torvalds 17811da177e4SLinus Torvalds if (tp->rcv_nxt == tp->copied_seq && 1782b03efcfbSDavid S. Miller !skb_queue_empty(&tp->ucopy.prequeue)) { 17831da177e4SLinus Torvalds do_prequeue: 17841da177e4SLinus Torvalds tcp_prequeue_process(sk); 17851da177e4SLinus Torvalds 17861da177e4SLinus Torvalds if ((chunk = len - tp->ucopy.len) != 0) { 1787ed88098eSPavel Emelyanov NET_ADD_STATS_USER(sock_net(sk), LINUX_MIB_TCPDIRECTCOPYFROMPREQUEUE, chunk); 17881da177e4SLinus Torvalds len -= chunk; 17891da177e4SLinus Torvalds copied += chunk; 17901da177e4SLinus Torvalds } 17911da177e4SLinus Torvalds } 17921da177e4SLinus Torvalds } 179377527313SIlpo Järvinen if ((flags & MSG_PEEK) && 179477527313SIlpo Järvinen (peek_seq - copied - urg_hole != tp->copied_seq)) { 1795e87cc472SJoe Perches net_dbg_ratelimited("TCP(%s:%d): Application bug, race in MSG_PEEK\n", 1796e87cc472SJoe Perches current->comm, 1797e87cc472SJoe Perches task_pid_nr(current)); 17981da177e4SLinus Torvalds peek_seq = tp->copied_seq; 17991da177e4SLinus Torvalds } 18001da177e4SLinus Torvalds continue; 18011da177e4SLinus Torvalds 18021da177e4SLinus Torvalds found_ok_skb: 18031da177e4SLinus Torvalds /* Ok so how much can we use? */ 18041da177e4SLinus Torvalds used = skb->len - offset; 18051da177e4SLinus Torvalds if (len < used) 18061da177e4SLinus Torvalds used = len; 18071da177e4SLinus Torvalds 18081da177e4SLinus Torvalds /* Do we have urgent data here? */ 18091da177e4SLinus Torvalds if (tp->urg_data) { 18101da177e4SLinus Torvalds u32 urg_offset = tp->urg_seq - *seq; 18111da177e4SLinus Torvalds if (urg_offset < used) { 18121da177e4SLinus Torvalds if (!urg_offset) { 18131da177e4SLinus Torvalds if (!sock_flag(sk, SOCK_URGINLINE)) { 18141da177e4SLinus Torvalds ++*seq; 181577527313SIlpo Järvinen urg_hole++; 18161da177e4SLinus Torvalds offset++; 18171da177e4SLinus Torvalds used--; 18181da177e4SLinus Torvalds if (!used) 18191da177e4SLinus Torvalds goto skip_copy; 18201da177e4SLinus Torvalds } 18211da177e4SLinus Torvalds } else 18221da177e4SLinus Torvalds used = urg_offset; 18231da177e4SLinus Torvalds } 18241da177e4SLinus Torvalds } 18251da177e4SLinus Torvalds 18261da177e4SLinus Torvalds if (!(flags & MSG_TRUNC)) { 18271a2449a8SChris Leech #ifdef CONFIG_NET_DMA 18281a2449a8SChris Leech if (!tp->ucopy.dma_chan && tp->ucopy.pinned_list) 1829a2bd1140SDave Jiang tp->ucopy.dma_chan = net_dma_find_channel(); 18301a2449a8SChris Leech 18311a2449a8SChris Leech if (tp->ucopy.dma_chan) { 18321a2449a8SChris Leech tp->ucopy.dma_cookie = dma_skb_copy_datagram_iovec( 18331a2449a8SChris Leech tp->ucopy.dma_chan, skb, offset, 18341a2449a8SChris Leech msg->msg_iov, used, 18351a2449a8SChris Leech tp->ucopy.pinned_list); 18361a2449a8SChris Leech 18371a2449a8SChris Leech if (tp->ucopy.dma_cookie < 0) { 18381a2449a8SChris Leech 1839afd46503SJoe Perches pr_alert("%s: dma_cookie < 0\n", 1840afd46503SJoe Perches __func__); 18411a2449a8SChris Leech 18421a2449a8SChris Leech /* Exception. Bailout! */ 18431a2449a8SChris Leech if (!copied) 18441a2449a8SChris Leech copied = -EFAULT; 18451a2449a8SChris Leech break; 18461a2449a8SChris Leech } 184773852e81SSteven J. Magnani 1848b9ee8683SBartlomiej Zolnierkiewicz dma_async_issue_pending(tp->ucopy.dma_chan); 184973852e81SSteven J. Magnani 18501a2449a8SChris Leech if ((offset + used) == skb->len) 1851dc6b9b78SEric Dumazet copied_early = true; 18521a2449a8SChris Leech 18531a2449a8SChris Leech } else 18541a2449a8SChris Leech #endif 18551a2449a8SChris Leech { 18561da177e4SLinus Torvalds err = skb_copy_datagram_iovec(skb, offset, 18571da177e4SLinus Torvalds msg->msg_iov, used); 18581da177e4SLinus Torvalds if (err) { 18591da177e4SLinus Torvalds /* Exception. Bailout! */ 18601da177e4SLinus Torvalds if (!copied) 18611da177e4SLinus Torvalds copied = -EFAULT; 18621da177e4SLinus Torvalds break; 18631da177e4SLinus Torvalds } 18641da177e4SLinus Torvalds } 18651a2449a8SChris Leech } 18661da177e4SLinus Torvalds 18671da177e4SLinus Torvalds *seq += used; 18681da177e4SLinus Torvalds copied += used; 18691da177e4SLinus Torvalds len -= used; 18701da177e4SLinus Torvalds 18711da177e4SLinus Torvalds tcp_rcv_space_adjust(sk); 18721da177e4SLinus Torvalds 18731da177e4SLinus Torvalds skip_copy: 18741da177e4SLinus Torvalds if (tp->urg_data && after(tp->copied_seq, tp->urg_seq)) { 18751da177e4SLinus Torvalds tp->urg_data = 0; 18769e412ba7SIlpo Järvinen tcp_fast_path_check(sk); 18771da177e4SLinus Torvalds } 18781da177e4SLinus Torvalds if (used + offset < skb->len) 18791da177e4SLinus Torvalds continue; 18801da177e4SLinus Torvalds 1881aa8223c7SArnaldo Carvalho de Melo if (tcp_hdr(skb)->fin) 18821da177e4SLinus Torvalds goto found_fin_ok; 18831a2449a8SChris Leech if (!(flags & MSG_PEEK)) { 18841a2449a8SChris Leech sk_eat_skb(sk, skb, copied_early); 1885dc6b9b78SEric Dumazet copied_early = false; 18861a2449a8SChris Leech } 18871da177e4SLinus Torvalds continue; 18881da177e4SLinus Torvalds 18891da177e4SLinus Torvalds found_fin_ok: 18901da177e4SLinus Torvalds /* Process the FIN. */ 18911da177e4SLinus Torvalds ++*seq; 18921a2449a8SChris Leech if (!(flags & MSG_PEEK)) { 18931a2449a8SChris Leech sk_eat_skb(sk, skb, copied_early); 1894dc6b9b78SEric Dumazet copied_early = false; 18951a2449a8SChris Leech } 18961da177e4SLinus Torvalds break; 18971da177e4SLinus Torvalds } while (len > 0); 18981da177e4SLinus Torvalds 18991da177e4SLinus Torvalds if (user_recv) { 1900b03efcfbSDavid S. Miller if (!skb_queue_empty(&tp->ucopy.prequeue)) { 19011da177e4SLinus Torvalds int chunk; 19021da177e4SLinus Torvalds 19031da177e4SLinus Torvalds tp->ucopy.len = copied > 0 ? len : 0; 19041da177e4SLinus Torvalds 19051da177e4SLinus Torvalds tcp_prequeue_process(sk); 19061da177e4SLinus Torvalds 19071da177e4SLinus Torvalds if (copied > 0 && (chunk = len - tp->ucopy.len) != 0) { 1908ed88098eSPavel Emelyanov NET_ADD_STATS_USER(sock_net(sk), LINUX_MIB_TCPDIRECTCOPYFROMPREQUEUE, chunk); 19091da177e4SLinus Torvalds len -= chunk; 19101da177e4SLinus Torvalds copied += chunk; 19111da177e4SLinus Torvalds } 19121da177e4SLinus Torvalds } 19131da177e4SLinus Torvalds 19141da177e4SLinus Torvalds tp->ucopy.task = NULL; 19151da177e4SLinus Torvalds tp->ucopy.len = 0; 19161da177e4SLinus Torvalds } 19171da177e4SLinus Torvalds 19181a2449a8SChris Leech #ifdef CONFIG_NET_DMA 191973852e81SSteven J. Magnani tcp_service_net_dma(sk, true); /* Wait for queue to drain */ 19201a2449a8SChris Leech tp->ucopy.dma_chan = NULL; 192173852e81SSteven J. Magnani 19221a2449a8SChris Leech if (tp->ucopy.pinned_list) { 19231a2449a8SChris Leech dma_unpin_iovec_pages(tp->ucopy.pinned_list); 19241a2449a8SChris Leech tp->ucopy.pinned_list = NULL; 19251a2449a8SChris Leech } 19261a2449a8SChris Leech #endif 19271a2449a8SChris Leech 19281da177e4SLinus Torvalds /* According to UNIX98, msg_name/msg_namelen are ignored 19291da177e4SLinus Torvalds * on connected socket. I was just happy when found this 8) --ANK 19301da177e4SLinus Torvalds */ 19311da177e4SLinus Torvalds 19321da177e4SLinus Torvalds /* Clean up data we have read: This will do ACK frames. */ 19330e4b4992SChris Leech tcp_cleanup_rbuf(sk, copied); 19341da177e4SLinus Torvalds 19351da177e4SLinus Torvalds release_sock(sk); 19361da177e4SLinus Torvalds return copied; 19371da177e4SLinus Torvalds 19381da177e4SLinus Torvalds out: 19391da177e4SLinus Torvalds release_sock(sk); 19401da177e4SLinus Torvalds return err; 19411da177e4SLinus Torvalds 19421da177e4SLinus Torvalds recv_urg: 1943377f0a08SRami Rosen err = tcp_recv_urg(sk, msg, len, flags); 19441da177e4SLinus Torvalds goto out; 1945c0e88ff0SPavel Emelyanov 1946c0e88ff0SPavel Emelyanov recv_sndq: 1947c0e88ff0SPavel Emelyanov err = tcp_peek_sndq(sk, msg, len); 1948c0e88ff0SPavel Emelyanov goto out; 19491da177e4SLinus Torvalds } 19504bc2f18bSEric Dumazet EXPORT_SYMBOL(tcp_recvmsg); 19511da177e4SLinus Torvalds 1952490d5046SIlpo Järvinen void tcp_set_state(struct sock *sk, int state) 1953490d5046SIlpo Järvinen { 1954490d5046SIlpo Järvinen int oldstate = sk->sk_state; 1955490d5046SIlpo Järvinen 1956490d5046SIlpo Järvinen switch (state) { 1957490d5046SIlpo Järvinen case TCP_ESTABLISHED: 1958490d5046SIlpo Järvinen if (oldstate != TCP_ESTABLISHED) 195981cc8a75SPavel Emelyanov TCP_INC_STATS(sock_net(sk), TCP_MIB_CURRESTAB); 1960490d5046SIlpo Järvinen break; 1961490d5046SIlpo Järvinen 1962490d5046SIlpo Järvinen case TCP_CLOSE: 1963490d5046SIlpo Järvinen if (oldstate == TCP_CLOSE_WAIT || oldstate == TCP_ESTABLISHED) 196481cc8a75SPavel Emelyanov TCP_INC_STATS(sock_net(sk), TCP_MIB_ESTABRESETS); 1965490d5046SIlpo Järvinen 1966490d5046SIlpo Järvinen sk->sk_prot->unhash(sk); 1967490d5046SIlpo Järvinen if (inet_csk(sk)->icsk_bind_hash && 1968490d5046SIlpo Järvinen !(sk->sk_userlocks & SOCK_BINDPORT_LOCK)) 1969ab1e0a13SArnaldo Carvalho de Melo inet_put_port(sk); 1970490d5046SIlpo Järvinen /* fall through */ 1971490d5046SIlpo Järvinen default: 1972490d5046SIlpo Järvinen if (oldstate == TCP_ESTABLISHED) 197374688e48SPavel Emelyanov TCP_DEC_STATS(sock_net(sk), TCP_MIB_CURRESTAB); 1974490d5046SIlpo Järvinen } 1975490d5046SIlpo Järvinen 1976490d5046SIlpo Järvinen /* Change state AFTER socket is unhashed to avoid closed 1977490d5046SIlpo Järvinen * socket sitting in hash tables. 1978490d5046SIlpo Järvinen */ 1979490d5046SIlpo Järvinen sk->sk_state = state; 1980490d5046SIlpo Järvinen 1981490d5046SIlpo Järvinen #ifdef STATE_TRACE 1982490d5046SIlpo Järvinen SOCK_DEBUG(sk, "TCP sk=%p, State %s -> %s\n", sk, statename[oldstate], statename[state]); 1983490d5046SIlpo Järvinen #endif 1984490d5046SIlpo Järvinen } 1985490d5046SIlpo Järvinen EXPORT_SYMBOL_GPL(tcp_set_state); 1986490d5046SIlpo Järvinen 19871da177e4SLinus Torvalds /* 19881da177e4SLinus Torvalds * State processing on a close. This implements the state shift for 19891da177e4SLinus Torvalds * sending our FIN frame. Note that we only send a FIN for some 19901da177e4SLinus Torvalds * states. A shutdown() may have already sent the FIN, or we may be 19911da177e4SLinus Torvalds * closed. 19921da177e4SLinus Torvalds */ 19931da177e4SLinus Torvalds 19949b5b5cffSArjan van de Ven static const unsigned char new_state[16] = { 19951da177e4SLinus Torvalds /* current state: new state: action: */ 19961da177e4SLinus Torvalds /* (Invalid) */ TCP_CLOSE, 19971da177e4SLinus Torvalds /* TCP_ESTABLISHED */ TCP_FIN_WAIT1 | TCP_ACTION_FIN, 19981da177e4SLinus Torvalds /* TCP_SYN_SENT */ TCP_CLOSE, 19991da177e4SLinus Torvalds /* TCP_SYN_RECV */ TCP_FIN_WAIT1 | TCP_ACTION_FIN, 20001da177e4SLinus Torvalds /* TCP_FIN_WAIT1 */ TCP_FIN_WAIT1, 20011da177e4SLinus Torvalds /* TCP_FIN_WAIT2 */ TCP_FIN_WAIT2, 20021da177e4SLinus Torvalds /* TCP_TIME_WAIT */ TCP_CLOSE, 20031da177e4SLinus Torvalds /* TCP_CLOSE */ TCP_CLOSE, 20041da177e4SLinus Torvalds /* TCP_CLOSE_WAIT */ TCP_LAST_ACK | TCP_ACTION_FIN, 20051da177e4SLinus Torvalds /* TCP_LAST_ACK */ TCP_LAST_ACK, 20061da177e4SLinus Torvalds /* TCP_LISTEN */ TCP_CLOSE, 20071da177e4SLinus Torvalds /* TCP_CLOSING */ TCP_CLOSING, 20081da177e4SLinus Torvalds }; 20091da177e4SLinus Torvalds 20101da177e4SLinus Torvalds static int tcp_close_state(struct sock *sk) 20111da177e4SLinus Torvalds { 20121da177e4SLinus Torvalds int next = (int)new_state[sk->sk_state]; 20131da177e4SLinus Torvalds int ns = next & TCP_STATE_MASK; 20141da177e4SLinus Torvalds 20151da177e4SLinus Torvalds tcp_set_state(sk, ns); 20161da177e4SLinus Torvalds 20171da177e4SLinus Torvalds return next & TCP_ACTION_FIN; 20181da177e4SLinus Torvalds } 20191da177e4SLinus Torvalds 20201da177e4SLinus Torvalds /* 20211da177e4SLinus Torvalds * Shutdown the sending side of a connection. Much like close except 20221f29b058SSatoru SATOH * that we don't receive shut down or sock_set_flag(sk, SOCK_DEAD). 20231da177e4SLinus Torvalds */ 20241da177e4SLinus Torvalds 20251da177e4SLinus Torvalds void tcp_shutdown(struct sock *sk, int how) 20261da177e4SLinus Torvalds { 20271da177e4SLinus Torvalds /* We need to grab some memory, and put together a FIN, 20281da177e4SLinus Torvalds * and then put it into the queue to be sent. 20291da177e4SLinus Torvalds * Tim MacKenzie(tym@dibbler.cs.monash.edu.au) 4 Dec '92. 20301da177e4SLinus Torvalds */ 20311da177e4SLinus Torvalds if (!(how & SEND_SHUTDOWN)) 20321da177e4SLinus Torvalds return; 20331da177e4SLinus Torvalds 20341da177e4SLinus Torvalds /* If we've already sent a FIN, or it's a closed state, skip this. */ 20351da177e4SLinus Torvalds if ((1 << sk->sk_state) & 20361da177e4SLinus Torvalds (TCPF_ESTABLISHED | TCPF_SYN_SENT | 20371da177e4SLinus Torvalds TCPF_SYN_RECV | TCPF_CLOSE_WAIT)) { 20381da177e4SLinus Torvalds /* Clear out any half completed packets. FIN if needed. */ 20391da177e4SLinus Torvalds if (tcp_close_state(sk)) 20401da177e4SLinus Torvalds tcp_send_fin(sk); 20411da177e4SLinus Torvalds } 20421da177e4SLinus Torvalds } 20434bc2f18bSEric Dumazet EXPORT_SYMBOL(tcp_shutdown); 20441da177e4SLinus Torvalds 2045efcdbf24SArun Sharma bool tcp_check_oom(struct sock *sk, int shift) 2046efcdbf24SArun Sharma { 2047efcdbf24SArun Sharma bool too_many_orphans, out_of_socket_memory; 2048efcdbf24SArun Sharma 2049efcdbf24SArun Sharma too_many_orphans = tcp_too_many_orphans(sk, shift); 2050efcdbf24SArun Sharma out_of_socket_memory = tcp_out_of_memory(sk); 2051efcdbf24SArun Sharma 2052e87cc472SJoe Perches if (too_many_orphans) 2053e87cc472SJoe Perches net_info_ratelimited("too many orphaned sockets\n"); 2054e87cc472SJoe Perches if (out_of_socket_memory) 2055e87cc472SJoe Perches net_info_ratelimited("out of memory -- consider tuning tcp_mem\n"); 2056efcdbf24SArun Sharma return too_many_orphans || out_of_socket_memory; 2057efcdbf24SArun Sharma } 2058efcdbf24SArun Sharma 20591da177e4SLinus Torvalds void tcp_close(struct sock *sk, long timeout) 20601da177e4SLinus Torvalds { 20611da177e4SLinus Torvalds struct sk_buff *skb; 20621da177e4SLinus Torvalds int data_was_unread = 0; 206375c2d907SHerbert Xu int state; 20641da177e4SLinus Torvalds 20651da177e4SLinus Torvalds lock_sock(sk); 20661da177e4SLinus Torvalds sk->sk_shutdown = SHUTDOWN_MASK; 20671da177e4SLinus Torvalds 20681da177e4SLinus Torvalds if (sk->sk_state == TCP_LISTEN) { 20691da177e4SLinus Torvalds tcp_set_state(sk, TCP_CLOSE); 20701da177e4SLinus Torvalds 20711da177e4SLinus Torvalds /* Special case. */ 20720a5578cfSArnaldo Carvalho de Melo inet_csk_listen_stop(sk); 20731da177e4SLinus Torvalds 20741da177e4SLinus Torvalds goto adjudge_to_death; 20751da177e4SLinus Torvalds } 20761da177e4SLinus Torvalds 20771da177e4SLinus Torvalds /* We need to flush the recv. buffs. We do this only on the 20781da177e4SLinus Torvalds * descriptor close, not protocol-sourced closes, because the 20791da177e4SLinus Torvalds * reader process may not have drained the data yet! 20801da177e4SLinus Torvalds */ 20811da177e4SLinus Torvalds while ((skb = __skb_dequeue(&sk->sk_receive_queue)) != NULL) { 20821da177e4SLinus Torvalds u32 len = TCP_SKB_CB(skb)->end_seq - TCP_SKB_CB(skb)->seq - 2083aa8223c7SArnaldo Carvalho de Melo tcp_hdr(skb)->fin; 20841da177e4SLinus Torvalds data_was_unread += len; 20851da177e4SLinus Torvalds __kfree_skb(skb); 20861da177e4SLinus Torvalds } 20871da177e4SLinus Torvalds 20883ab224beSHideo Aoki sk_mem_reclaim(sk); 20891da177e4SLinus Torvalds 2090565b7b2dSKonstantin Khorenko /* If socket has been already reset (e.g. in tcp_reset()) - kill it. */ 2091565b7b2dSKonstantin Khorenko if (sk->sk_state == TCP_CLOSE) 2092565b7b2dSKonstantin Khorenko goto adjudge_to_death; 2093565b7b2dSKonstantin Khorenko 209465bb723cSGerrit Renker /* As outlined in RFC 2525, section 2.17, we send a RST here because 209565bb723cSGerrit Renker * data was lost. To witness the awful effects of the old behavior of 209665bb723cSGerrit Renker * always doing a FIN, run an older 2.1.x kernel or 2.0.x, start a bulk 209765bb723cSGerrit Renker * GET in an FTP client, suspend the process, wait for the client to 209865bb723cSGerrit Renker * advertise a zero window, then kill -9 the FTP client, wheee... 209965bb723cSGerrit Renker * Note: timeout is always zero in such a case. 21001da177e4SLinus Torvalds */ 2101ee995283SPavel Emelyanov if (unlikely(tcp_sk(sk)->repair)) { 2102ee995283SPavel Emelyanov sk->sk_prot->disconnect(sk, 0); 2103ee995283SPavel Emelyanov } else if (data_was_unread) { 21041da177e4SLinus Torvalds /* Unread data was tossed, zap the connection. */ 21056f67c817SPavel Emelyanov NET_INC_STATS_USER(sock_net(sk), LINUX_MIB_TCPABORTONCLOSE); 21061da177e4SLinus Torvalds tcp_set_state(sk, TCP_CLOSE); 2107aa133076SWu Fengguang tcp_send_active_reset(sk, sk->sk_allocation); 21081da177e4SLinus Torvalds } else if (sock_flag(sk, SOCK_LINGER) && !sk->sk_lingertime) { 21091da177e4SLinus Torvalds /* Check zero linger _after_ checking for unread data. */ 21101da177e4SLinus Torvalds sk->sk_prot->disconnect(sk, 0); 21116f67c817SPavel Emelyanov NET_INC_STATS_USER(sock_net(sk), LINUX_MIB_TCPABORTONDATA); 21121da177e4SLinus Torvalds } else if (tcp_close_state(sk)) { 21131da177e4SLinus Torvalds /* We FIN if the application ate all the data before 21141da177e4SLinus Torvalds * zapping the connection. 21151da177e4SLinus Torvalds */ 21161da177e4SLinus Torvalds 21171da177e4SLinus Torvalds /* RED-PEN. Formally speaking, we have broken TCP state 21181da177e4SLinus Torvalds * machine. State transitions: 21191da177e4SLinus Torvalds * 21201da177e4SLinus Torvalds * TCP_ESTABLISHED -> TCP_FIN_WAIT1 21211da177e4SLinus Torvalds * TCP_SYN_RECV -> TCP_FIN_WAIT1 (forget it, it's impossible) 21221da177e4SLinus Torvalds * TCP_CLOSE_WAIT -> TCP_LAST_ACK 21231da177e4SLinus Torvalds * 21241da177e4SLinus Torvalds * are legal only when FIN has been sent (i.e. in window), 21251da177e4SLinus Torvalds * rather than queued out of window. Purists blame. 21261da177e4SLinus Torvalds * 21271da177e4SLinus Torvalds * F.e. "RFC state" is ESTABLISHED, 21281da177e4SLinus Torvalds * if Linux state is FIN-WAIT-1, but FIN is still not sent. 21291da177e4SLinus Torvalds * 21301da177e4SLinus Torvalds * The visible declinations are that sometimes 21311da177e4SLinus Torvalds * we enter time-wait state, when it is not required really 21321da177e4SLinus Torvalds * (harmless), do not send active resets, when they are 21331da177e4SLinus Torvalds * required by specs (TCP_ESTABLISHED, TCP_CLOSE_WAIT, when 21341da177e4SLinus Torvalds * they look as CLOSING or LAST_ACK for Linux) 21351da177e4SLinus Torvalds * Probably, I missed some more holelets. 21361da177e4SLinus Torvalds * --ANK 21378336886fSJerry Chu * XXX (TFO) - To start off we don't support SYN+ACK+FIN 21388336886fSJerry Chu * in a single packet! (May consider it later but will 21398336886fSJerry Chu * probably need API support or TCP_CORK SYN-ACK until 21408336886fSJerry Chu * data is written and socket is closed.) 21411da177e4SLinus Torvalds */ 21421da177e4SLinus Torvalds tcp_send_fin(sk); 21431da177e4SLinus Torvalds } 21441da177e4SLinus Torvalds 21451da177e4SLinus Torvalds sk_stream_wait_close(sk, timeout); 21461da177e4SLinus Torvalds 21471da177e4SLinus Torvalds adjudge_to_death: 214875c2d907SHerbert Xu state = sk->sk_state; 214975c2d907SHerbert Xu sock_hold(sk); 215075c2d907SHerbert Xu sock_orphan(sk); 215175c2d907SHerbert Xu 21521da177e4SLinus Torvalds /* It is the last release_sock in its life. It will remove backlog. */ 21531da177e4SLinus Torvalds release_sock(sk); 21541da177e4SLinus Torvalds 21551da177e4SLinus Torvalds 21561da177e4SLinus Torvalds /* Now socket is owned by kernel and we acquire BH lock 21571da177e4SLinus Torvalds to finish close. No need to check for user refs. 21581da177e4SLinus Torvalds */ 21591da177e4SLinus Torvalds local_bh_disable(); 21601da177e4SLinus Torvalds bh_lock_sock(sk); 2161547b792cSIlpo Järvinen WARN_ON(sock_owned_by_user(sk)); 21621da177e4SLinus Torvalds 2163eb4dea58SHerbert Xu percpu_counter_inc(sk->sk_prot->orphan_count); 2164eb4dea58SHerbert Xu 216575c2d907SHerbert Xu /* Have we already been destroyed by a softirq or backlog? */ 216675c2d907SHerbert Xu if (state != TCP_CLOSE && sk->sk_state == TCP_CLOSE) 216775c2d907SHerbert Xu goto out; 21681da177e4SLinus Torvalds 21691da177e4SLinus Torvalds /* This is a (useful) BSD violating of the RFC. There is a 21701da177e4SLinus Torvalds * problem with TCP as specified in that the other end could 21711da177e4SLinus Torvalds * keep a socket open forever with no application left this end. 21721da177e4SLinus Torvalds * We use a 3 minute timeout (about the same as BSD) then kill 21731da177e4SLinus Torvalds * our end. If they send after that then tough - BUT: long enough 21741da177e4SLinus Torvalds * that we won't make the old 4*rto = almost no time - whoops 21751da177e4SLinus Torvalds * reset mistake. 21761da177e4SLinus Torvalds * 21771da177e4SLinus Torvalds * Nope, it was not mistake. It is really desired behaviour 21781da177e4SLinus Torvalds * f.e. on http servers, when such sockets are useless, but 21791da177e4SLinus Torvalds * consume significant resources. Let's do it with special 21801da177e4SLinus Torvalds * linger2 option. --ANK 21811da177e4SLinus Torvalds */ 21821da177e4SLinus Torvalds 21831da177e4SLinus Torvalds if (sk->sk_state == TCP_FIN_WAIT2) { 21841da177e4SLinus Torvalds struct tcp_sock *tp = tcp_sk(sk); 21851da177e4SLinus Torvalds if (tp->linger2 < 0) { 21861da177e4SLinus Torvalds tcp_set_state(sk, TCP_CLOSE); 21871da177e4SLinus Torvalds tcp_send_active_reset(sk, GFP_ATOMIC); 2188de0744afSPavel Emelyanov NET_INC_STATS_BH(sock_net(sk), 2189de0744afSPavel Emelyanov LINUX_MIB_TCPABORTONLINGER); 21901da177e4SLinus Torvalds } else { 2191463c84b9SArnaldo Carvalho de Melo const int tmo = tcp_fin_time(sk); 21921da177e4SLinus Torvalds 21931da177e4SLinus Torvalds if (tmo > TCP_TIMEWAIT_LEN) { 219452499afeSDavid S. Miller inet_csk_reset_keepalive_timer(sk, 219552499afeSDavid S. Miller tmo - TCP_TIMEWAIT_LEN); 21961da177e4SLinus Torvalds } else { 21971da177e4SLinus Torvalds tcp_time_wait(sk, TCP_FIN_WAIT2, tmo); 21981da177e4SLinus Torvalds goto out; 21991da177e4SLinus Torvalds } 22001da177e4SLinus Torvalds } 22011da177e4SLinus Torvalds } 22021da177e4SLinus Torvalds if (sk->sk_state != TCP_CLOSE) { 22033ab224beSHideo Aoki sk_mem_reclaim(sk); 2204efcdbf24SArun Sharma if (tcp_check_oom(sk, 0)) { 22051da177e4SLinus Torvalds tcp_set_state(sk, TCP_CLOSE); 22061da177e4SLinus Torvalds tcp_send_active_reset(sk, GFP_ATOMIC); 2207de0744afSPavel Emelyanov NET_INC_STATS_BH(sock_net(sk), 2208de0744afSPavel Emelyanov LINUX_MIB_TCPABORTONMEMORY); 22091da177e4SLinus Torvalds } 22101da177e4SLinus Torvalds } 22111da177e4SLinus Torvalds 22128336886fSJerry Chu if (sk->sk_state == TCP_CLOSE) { 22138336886fSJerry Chu struct request_sock *req = tcp_sk(sk)->fastopen_rsk; 22148336886fSJerry Chu /* We could get here with a non-NULL req if the socket is 22158336886fSJerry Chu * aborted (e.g., closed with unread data) before 3WHS 22168336886fSJerry Chu * finishes. 22178336886fSJerry Chu */ 22188336886fSJerry Chu if (req != NULL) 22198336886fSJerry Chu reqsk_fastopen_remove(sk, req, false); 22200a5578cfSArnaldo Carvalho de Melo inet_csk_destroy_sock(sk); 22218336886fSJerry Chu } 22221da177e4SLinus Torvalds /* Otherwise, socket is reprieved until protocol close. */ 22231da177e4SLinus Torvalds 22241da177e4SLinus Torvalds out: 22251da177e4SLinus Torvalds bh_unlock_sock(sk); 22261da177e4SLinus Torvalds local_bh_enable(); 22271da177e4SLinus Torvalds sock_put(sk); 22281da177e4SLinus Torvalds } 22294bc2f18bSEric Dumazet EXPORT_SYMBOL(tcp_close); 22301da177e4SLinus Torvalds 22311da177e4SLinus Torvalds /* These states need RST on ABORT according to RFC793 */ 22321da177e4SLinus Torvalds 2233a2a385d6SEric Dumazet static inline bool tcp_need_reset(int state) 22341da177e4SLinus Torvalds { 22351da177e4SLinus Torvalds return (1 << state) & 22361da177e4SLinus Torvalds (TCPF_ESTABLISHED | TCPF_CLOSE_WAIT | TCPF_FIN_WAIT1 | 22371da177e4SLinus Torvalds TCPF_FIN_WAIT2 | TCPF_SYN_RECV); 22381da177e4SLinus Torvalds } 22391da177e4SLinus Torvalds 22401da177e4SLinus Torvalds int tcp_disconnect(struct sock *sk, int flags) 22411da177e4SLinus Torvalds { 22421da177e4SLinus Torvalds struct inet_sock *inet = inet_sk(sk); 2243463c84b9SArnaldo Carvalho de Melo struct inet_connection_sock *icsk = inet_csk(sk); 22441da177e4SLinus Torvalds struct tcp_sock *tp = tcp_sk(sk); 22451da177e4SLinus Torvalds int err = 0; 22461da177e4SLinus Torvalds int old_state = sk->sk_state; 22471da177e4SLinus Torvalds 22481da177e4SLinus Torvalds if (old_state != TCP_CLOSE) 22491da177e4SLinus Torvalds tcp_set_state(sk, TCP_CLOSE); 22501da177e4SLinus Torvalds 22511da177e4SLinus Torvalds /* ABORT function of RFC793 */ 22521da177e4SLinus Torvalds if (old_state == TCP_LISTEN) { 22530a5578cfSArnaldo Carvalho de Melo inet_csk_listen_stop(sk); 2254ee995283SPavel Emelyanov } else if (unlikely(tp->repair)) { 2255ee995283SPavel Emelyanov sk->sk_err = ECONNABORTED; 22561da177e4SLinus Torvalds } else if (tcp_need_reset(old_state) || 22571da177e4SLinus Torvalds (tp->snd_nxt != tp->write_seq && 22581da177e4SLinus Torvalds (1 << old_state) & (TCPF_CLOSING | TCPF_LAST_ACK))) { 2259caa20d9aSStephen Hemminger /* The last check adjusts for discrepancy of Linux wrt. RFC 22601da177e4SLinus Torvalds * states 22611da177e4SLinus Torvalds */ 22621da177e4SLinus Torvalds tcp_send_active_reset(sk, gfp_any()); 22631da177e4SLinus Torvalds sk->sk_err = ECONNRESET; 22641da177e4SLinus Torvalds } else if (old_state == TCP_SYN_SENT) 22651da177e4SLinus Torvalds sk->sk_err = ECONNRESET; 22661da177e4SLinus Torvalds 22671da177e4SLinus Torvalds tcp_clear_xmit_timers(sk); 22681da177e4SLinus Torvalds __skb_queue_purge(&sk->sk_receive_queue); 2269fe067e8aSDavid S. Miller tcp_write_queue_purge(sk); 22701da177e4SLinus Torvalds __skb_queue_purge(&tp->out_of_order_queue); 22711a2449a8SChris Leech #ifdef CONFIG_NET_DMA 22721a2449a8SChris Leech __skb_queue_purge(&sk->sk_async_wait_queue); 22731a2449a8SChris Leech #endif 22741da177e4SLinus Torvalds 2275c720c7e8SEric Dumazet inet->inet_dport = 0; 22761da177e4SLinus Torvalds 22771da177e4SLinus Torvalds if (!(sk->sk_userlocks & SOCK_BINDADDR_LOCK)) 22781da177e4SLinus Torvalds inet_reset_saddr(sk); 22791da177e4SLinus Torvalds 22801da177e4SLinus Torvalds sk->sk_shutdown = 0; 22811da177e4SLinus Torvalds sock_reset_flag(sk, SOCK_DONE); 22821da177e4SLinus Torvalds tp->srtt = 0; 22831da177e4SLinus Torvalds if ((tp->write_seq += tp->max_window + 2) == 0) 22841da177e4SLinus Torvalds tp->write_seq = 1; 2285463c84b9SArnaldo Carvalho de Melo icsk->icsk_backoff = 0; 22861da177e4SLinus Torvalds tp->snd_cwnd = 2; 22876687e988SArnaldo Carvalho de Melo icsk->icsk_probes_out = 0; 22881da177e4SLinus Torvalds tp->packets_out = 0; 22890b6a05c1SIlpo Järvinen tp->snd_ssthresh = TCP_INFINITE_SSTHRESH; 22901da177e4SLinus Torvalds tp->snd_cwnd_cnt = 0; 22911fdf475aSEric Dumazet tp->window_clamp = 0; 22926687e988SArnaldo Carvalho de Melo tcp_set_ca_state(sk, TCP_CA_Open); 22931da177e4SLinus Torvalds tcp_clear_retrans(tp); 2294463c84b9SArnaldo Carvalho de Melo inet_csk_delack_init(sk); 2295fe067e8aSDavid S. Miller tcp_init_send_head(sk); 2296b40b4f79SSrinivas Aji memset(&tp->rx_opt, 0, sizeof(tp->rx_opt)); 22971da177e4SLinus Torvalds __sk_dst_reset(sk); 22981da177e4SLinus Torvalds 2299c720c7e8SEric Dumazet WARN_ON(inet->inet_num && !icsk->icsk_bind_hash); 23001da177e4SLinus Torvalds 23011da177e4SLinus Torvalds sk->sk_error_report(sk); 23021da177e4SLinus Torvalds return err; 23031da177e4SLinus Torvalds } 23044bc2f18bSEric Dumazet EXPORT_SYMBOL(tcp_disconnect); 23051da177e4SLinus Torvalds 2306bb68b647SChristoph Paasch void tcp_sock_destruct(struct sock *sk) 2307bb68b647SChristoph Paasch { 2308bb68b647SChristoph Paasch inet_sock_destruct(sk); 2309bb68b647SChristoph Paasch 2310bb68b647SChristoph Paasch kfree(inet_csk(sk)->icsk_accept_queue.fastopenq); 2311bb68b647SChristoph Paasch } 2312bb68b647SChristoph Paasch 2313a2a385d6SEric Dumazet static inline bool tcp_can_repair_sock(const struct sock *sk) 2314ee995283SPavel Emelyanov { 231552e804c6SEric W. Biederman return ns_capable(sock_net(sk)->user_ns, CAP_NET_ADMIN) && 2316ee995283SPavel Emelyanov ((1 << sk->sk_state) & (TCPF_CLOSE | TCPF_ESTABLISHED)); 2317ee995283SPavel Emelyanov } 2318ee995283SPavel Emelyanov 2319de248a75SPavel Emelyanov static int tcp_repair_options_est(struct tcp_sock *tp, 2320de248a75SPavel Emelyanov struct tcp_repair_opt __user *optbuf, unsigned int len) 2321b139ba4eSPavel Emelyanov { 2322de248a75SPavel Emelyanov struct tcp_repair_opt opt; 2323b139ba4eSPavel Emelyanov 2324de248a75SPavel Emelyanov while (len >= sizeof(opt)) { 2325de248a75SPavel Emelyanov if (copy_from_user(&opt, optbuf, sizeof(opt))) 2326b139ba4eSPavel Emelyanov return -EFAULT; 2327b139ba4eSPavel Emelyanov 2328b139ba4eSPavel Emelyanov optbuf++; 2329de248a75SPavel Emelyanov len -= sizeof(opt); 2330b139ba4eSPavel Emelyanov 2331de248a75SPavel Emelyanov switch (opt.opt_code) { 2332de248a75SPavel Emelyanov case TCPOPT_MSS: 2333de248a75SPavel Emelyanov tp->rx_opt.mss_clamp = opt.opt_val; 2334b139ba4eSPavel Emelyanov break; 2335de248a75SPavel Emelyanov case TCPOPT_WINDOW: 2336bc26ccd8SAndrey Vagin { 2337bc26ccd8SAndrey Vagin u16 snd_wscale = opt.opt_val & 0xFFFF; 2338bc26ccd8SAndrey Vagin u16 rcv_wscale = opt.opt_val >> 16; 2339bc26ccd8SAndrey Vagin 2340bc26ccd8SAndrey Vagin if (snd_wscale > 14 || rcv_wscale > 14) 2341b139ba4eSPavel Emelyanov return -EFBIG; 2342b139ba4eSPavel Emelyanov 2343bc26ccd8SAndrey Vagin tp->rx_opt.snd_wscale = snd_wscale; 2344bc26ccd8SAndrey Vagin tp->rx_opt.rcv_wscale = rcv_wscale; 2345bc26ccd8SAndrey Vagin tp->rx_opt.wscale_ok = 1; 2346bc26ccd8SAndrey Vagin } 2347b139ba4eSPavel Emelyanov break; 2348b139ba4eSPavel Emelyanov case TCPOPT_SACK_PERM: 2349de248a75SPavel Emelyanov if (opt.opt_val != 0) 2350de248a75SPavel Emelyanov return -EINVAL; 2351de248a75SPavel Emelyanov 2352b139ba4eSPavel Emelyanov tp->rx_opt.sack_ok |= TCP_SACK_SEEN; 2353b139ba4eSPavel Emelyanov if (sysctl_tcp_fack) 2354b139ba4eSPavel Emelyanov tcp_enable_fack(tp); 2355b139ba4eSPavel Emelyanov break; 2356b139ba4eSPavel Emelyanov case TCPOPT_TIMESTAMP: 2357de248a75SPavel Emelyanov if (opt.opt_val != 0) 2358de248a75SPavel Emelyanov return -EINVAL; 2359de248a75SPavel Emelyanov 2360b139ba4eSPavel Emelyanov tp->rx_opt.tstamp_ok = 1; 2361b139ba4eSPavel Emelyanov break; 2362b139ba4eSPavel Emelyanov } 2363b139ba4eSPavel Emelyanov } 2364b139ba4eSPavel Emelyanov 2365b139ba4eSPavel Emelyanov return 0; 2366b139ba4eSPavel Emelyanov } 2367b139ba4eSPavel Emelyanov 23681da177e4SLinus Torvalds /* 23691da177e4SLinus Torvalds * Socket option code for TCP. 23701da177e4SLinus Torvalds */ 23713fdadf7dSDmitry Mishin static int do_tcp_setsockopt(struct sock *sk, int level, 2372b7058842SDavid S. Miller int optname, char __user *optval, unsigned int optlen) 23731da177e4SLinus Torvalds { 23741da177e4SLinus Torvalds struct tcp_sock *tp = tcp_sk(sk); 2375463c84b9SArnaldo Carvalho de Melo struct inet_connection_sock *icsk = inet_csk(sk); 23761da177e4SLinus Torvalds int val; 23771da177e4SLinus Torvalds int err = 0; 23781da177e4SLinus Torvalds 2379e56fb50fSWilliam Allen Simpson /* These are data/string values, all the others are ints */ 2380e56fb50fSWilliam Allen Simpson switch (optname) { 2381e56fb50fSWilliam Allen Simpson case TCP_CONGESTION: { 23825f8ef48dSStephen Hemminger char name[TCP_CA_NAME_MAX]; 23835f8ef48dSStephen Hemminger 23845f8ef48dSStephen Hemminger if (optlen < 1) 23855f8ef48dSStephen Hemminger return -EINVAL; 23865f8ef48dSStephen Hemminger 23875f8ef48dSStephen Hemminger val = strncpy_from_user(name, optval, 23884fdb78d3SAndrew Morton min_t(long, TCP_CA_NAME_MAX-1, optlen)); 23895f8ef48dSStephen Hemminger if (val < 0) 23905f8ef48dSStephen Hemminger return -EFAULT; 23915f8ef48dSStephen Hemminger name[val] = 0; 23925f8ef48dSStephen Hemminger 23935f8ef48dSStephen Hemminger lock_sock(sk); 23946687e988SArnaldo Carvalho de Melo err = tcp_set_congestion_control(sk, name); 23955f8ef48dSStephen Hemminger release_sock(sk); 23965f8ef48dSStephen Hemminger return err; 23975f8ef48dSStephen Hemminger } 2398e56fb50fSWilliam Allen Simpson default: 2399e56fb50fSWilliam Allen Simpson /* fallthru */ 2400e56fb50fSWilliam Allen Simpson break; 2401ccbd6a5aSJoe Perches } 24025f8ef48dSStephen Hemminger 24031da177e4SLinus Torvalds if (optlen < sizeof(int)) 24041da177e4SLinus Torvalds return -EINVAL; 24051da177e4SLinus Torvalds 24061da177e4SLinus Torvalds if (get_user(val, (int __user *)optval)) 24071da177e4SLinus Torvalds return -EFAULT; 24081da177e4SLinus Torvalds 24091da177e4SLinus Torvalds lock_sock(sk); 24101da177e4SLinus Torvalds 24111da177e4SLinus Torvalds switch (optname) { 24121da177e4SLinus Torvalds case TCP_MAXSEG: 24131da177e4SLinus Torvalds /* Values greater than interface MTU won't take effect. However 24141da177e4SLinus Torvalds * at the point when this call is done we typically don't yet 24151da177e4SLinus Torvalds * know which interface is going to be used */ 2416c39508d6SDavid S. Miller if (val < TCP_MIN_MSS || val > MAX_TCP_WINDOW) { 24171da177e4SLinus Torvalds err = -EINVAL; 24181da177e4SLinus Torvalds break; 24191da177e4SLinus Torvalds } 24201da177e4SLinus Torvalds tp->rx_opt.user_mss = val; 24211da177e4SLinus Torvalds break; 24221da177e4SLinus Torvalds 24231da177e4SLinus Torvalds case TCP_NODELAY: 24241da177e4SLinus Torvalds if (val) { 24251da177e4SLinus Torvalds /* TCP_NODELAY is weaker than TCP_CORK, so that 24261da177e4SLinus Torvalds * this option on corked socket is remembered, but 24271da177e4SLinus Torvalds * it is not activated until cork is cleared. 24281da177e4SLinus Torvalds * 24291da177e4SLinus Torvalds * However, when TCP_NODELAY is set we make 24301da177e4SLinus Torvalds * an explicit push, which overrides even TCP_CORK 24311da177e4SLinus Torvalds * for currently queued segments. 24321da177e4SLinus Torvalds */ 24331da177e4SLinus Torvalds tp->nonagle |= TCP_NAGLE_OFF|TCP_NAGLE_PUSH; 24349e412ba7SIlpo Järvinen tcp_push_pending_frames(sk); 24351da177e4SLinus Torvalds } else { 24361da177e4SLinus Torvalds tp->nonagle &= ~TCP_NAGLE_OFF; 24371da177e4SLinus Torvalds } 24381da177e4SLinus Torvalds break; 24391da177e4SLinus Torvalds 244036e31b0aSAndreas Petlund case TCP_THIN_LINEAR_TIMEOUTS: 244136e31b0aSAndreas Petlund if (val < 0 || val > 1) 244236e31b0aSAndreas Petlund err = -EINVAL; 244336e31b0aSAndreas Petlund else 244436e31b0aSAndreas Petlund tp->thin_lto = val; 244536e31b0aSAndreas Petlund break; 244636e31b0aSAndreas Petlund 24477e380175SAndreas Petlund case TCP_THIN_DUPACK: 24487e380175SAndreas Petlund if (val < 0 || val > 1) 24497e380175SAndreas Petlund err = -EINVAL; 24507e380175SAndreas Petlund else 24517e380175SAndreas Petlund tp->thin_dupack = val; 2452eed530b6SYuchung Cheng if (tp->thin_dupack) 2453eed530b6SYuchung Cheng tcp_disable_early_retrans(tp); 24547e380175SAndreas Petlund break; 24557e380175SAndreas Petlund 2456ee995283SPavel Emelyanov case TCP_REPAIR: 2457ee995283SPavel Emelyanov if (!tcp_can_repair_sock(sk)) 2458ee995283SPavel Emelyanov err = -EPERM; 2459ee995283SPavel Emelyanov else if (val == 1) { 2460ee995283SPavel Emelyanov tp->repair = 1; 2461ee995283SPavel Emelyanov sk->sk_reuse = SK_FORCE_REUSE; 2462ee995283SPavel Emelyanov tp->repair_queue = TCP_NO_QUEUE; 2463ee995283SPavel Emelyanov } else if (val == 0) { 2464ee995283SPavel Emelyanov tp->repair = 0; 2465ee995283SPavel Emelyanov sk->sk_reuse = SK_NO_REUSE; 2466ee995283SPavel Emelyanov tcp_send_window_probe(sk); 2467ee995283SPavel Emelyanov } else 2468ee995283SPavel Emelyanov err = -EINVAL; 2469ee995283SPavel Emelyanov 2470ee995283SPavel Emelyanov break; 2471ee995283SPavel Emelyanov 2472ee995283SPavel Emelyanov case TCP_REPAIR_QUEUE: 2473ee995283SPavel Emelyanov if (!tp->repair) 2474ee995283SPavel Emelyanov err = -EPERM; 2475ee995283SPavel Emelyanov else if (val < TCP_QUEUES_NR) 2476ee995283SPavel Emelyanov tp->repair_queue = val; 2477ee995283SPavel Emelyanov else 2478ee995283SPavel Emelyanov err = -EINVAL; 2479ee995283SPavel Emelyanov break; 2480ee995283SPavel Emelyanov 2481ee995283SPavel Emelyanov case TCP_QUEUE_SEQ: 2482ee995283SPavel Emelyanov if (sk->sk_state != TCP_CLOSE) 2483ee995283SPavel Emelyanov err = -EPERM; 2484ee995283SPavel Emelyanov else if (tp->repair_queue == TCP_SEND_QUEUE) 2485ee995283SPavel Emelyanov tp->write_seq = val; 2486ee995283SPavel Emelyanov else if (tp->repair_queue == TCP_RECV_QUEUE) 2487ee995283SPavel Emelyanov tp->rcv_nxt = val; 2488ee995283SPavel Emelyanov else 2489ee995283SPavel Emelyanov err = -EINVAL; 2490ee995283SPavel Emelyanov break; 2491ee995283SPavel Emelyanov 2492b139ba4eSPavel Emelyanov case TCP_REPAIR_OPTIONS: 2493b139ba4eSPavel Emelyanov if (!tp->repair) 2494b139ba4eSPavel Emelyanov err = -EINVAL; 2495b139ba4eSPavel Emelyanov else if (sk->sk_state == TCP_ESTABLISHED) 2496de248a75SPavel Emelyanov err = tcp_repair_options_est(tp, 2497de248a75SPavel Emelyanov (struct tcp_repair_opt __user *)optval, 2498de248a75SPavel Emelyanov optlen); 2499b139ba4eSPavel Emelyanov else 2500b139ba4eSPavel Emelyanov err = -EPERM; 2501b139ba4eSPavel Emelyanov break; 2502b139ba4eSPavel Emelyanov 25031da177e4SLinus Torvalds case TCP_CORK: 25041da177e4SLinus Torvalds /* When set indicates to always queue non-full frames. 25051da177e4SLinus Torvalds * Later the user clears this option and we transmit 25061da177e4SLinus Torvalds * any pending partial frames in the queue. This is 25071da177e4SLinus Torvalds * meant to be used alongside sendfile() to get properly 25081da177e4SLinus Torvalds * filled frames when the user (for example) must write 25091da177e4SLinus Torvalds * out headers with a write() call first and then use 25101da177e4SLinus Torvalds * sendfile to send out the data parts. 25111da177e4SLinus Torvalds * 25121da177e4SLinus Torvalds * TCP_CORK can be set together with TCP_NODELAY and it is 25131da177e4SLinus Torvalds * stronger than TCP_NODELAY. 25141da177e4SLinus Torvalds */ 25151da177e4SLinus Torvalds if (val) { 25161da177e4SLinus Torvalds tp->nonagle |= TCP_NAGLE_CORK; 25171da177e4SLinus Torvalds } else { 25181da177e4SLinus Torvalds tp->nonagle &= ~TCP_NAGLE_CORK; 25191da177e4SLinus Torvalds if (tp->nonagle&TCP_NAGLE_OFF) 25201da177e4SLinus Torvalds tp->nonagle |= TCP_NAGLE_PUSH; 25219e412ba7SIlpo Järvinen tcp_push_pending_frames(sk); 25221da177e4SLinus Torvalds } 25231da177e4SLinus Torvalds break; 25241da177e4SLinus Torvalds 25251da177e4SLinus Torvalds case TCP_KEEPIDLE: 25261da177e4SLinus Torvalds if (val < 1 || val > MAX_TCP_KEEPIDLE) 25271da177e4SLinus Torvalds err = -EINVAL; 25281da177e4SLinus Torvalds else { 25291da177e4SLinus Torvalds tp->keepalive_time = val * HZ; 25301da177e4SLinus Torvalds if (sock_flag(sk, SOCK_KEEPOPEN) && 25311da177e4SLinus Torvalds !((1 << sk->sk_state) & 25321da177e4SLinus Torvalds (TCPF_CLOSE | TCPF_LISTEN))) { 25336c37e5deSFlavio Leitner u32 elapsed = keepalive_time_elapsed(tp); 25341da177e4SLinus Torvalds if (tp->keepalive_time > elapsed) 25351da177e4SLinus Torvalds elapsed = tp->keepalive_time - elapsed; 25361da177e4SLinus Torvalds else 25371da177e4SLinus Torvalds elapsed = 0; 2538463c84b9SArnaldo Carvalho de Melo inet_csk_reset_keepalive_timer(sk, elapsed); 25391da177e4SLinus Torvalds } 25401da177e4SLinus Torvalds } 25411da177e4SLinus Torvalds break; 25421da177e4SLinus Torvalds case TCP_KEEPINTVL: 25431da177e4SLinus Torvalds if (val < 1 || val > MAX_TCP_KEEPINTVL) 25441da177e4SLinus Torvalds err = -EINVAL; 25451da177e4SLinus Torvalds else 25461da177e4SLinus Torvalds tp->keepalive_intvl = val * HZ; 25471da177e4SLinus Torvalds break; 25481da177e4SLinus Torvalds case TCP_KEEPCNT: 25491da177e4SLinus Torvalds if (val < 1 || val > MAX_TCP_KEEPCNT) 25501da177e4SLinus Torvalds err = -EINVAL; 25511da177e4SLinus Torvalds else 25521da177e4SLinus Torvalds tp->keepalive_probes = val; 25531da177e4SLinus Torvalds break; 25541da177e4SLinus Torvalds case TCP_SYNCNT: 25551da177e4SLinus Torvalds if (val < 1 || val > MAX_TCP_SYNCNT) 25561da177e4SLinus Torvalds err = -EINVAL; 25571da177e4SLinus Torvalds else 2558463c84b9SArnaldo Carvalho de Melo icsk->icsk_syn_retries = val; 25591da177e4SLinus Torvalds break; 25601da177e4SLinus Torvalds 25611da177e4SLinus Torvalds case TCP_LINGER2: 25621da177e4SLinus Torvalds if (val < 0) 25631da177e4SLinus Torvalds tp->linger2 = -1; 25641da177e4SLinus Torvalds else if (val > sysctl_tcp_fin_timeout / HZ) 25651da177e4SLinus Torvalds tp->linger2 = 0; 25661da177e4SLinus Torvalds else 25671da177e4SLinus Torvalds tp->linger2 = val * HZ; 25681da177e4SLinus Torvalds break; 25691da177e4SLinus Torvalds 25701da177e4SLinus Torvalds case TCP_DEFER_ACCEPT: 2571b103cf34SJulian Anastasov /* Translate value in seconds to number of retransmits */ 2572b103cf34SJulian Anastasov icsk->icsk_accept_queue.rskq_defer_accept = 2573b103cf34SJulian Anastasov secs_to_retrans(val, TCP_TIMEOUT_INIT / HZ, 2574b103cf34SJulian Anastasov TCP_RTO_MAX / HZ); 25751da177e4SLinus Torvalds break; 25761da177e4SLinus Torvalds 25771da177e4SLinus Torvalds case TCP_WINDOW_CLAMP: 25781da177e4SLinus Torvalds if (!val) { 25791da177e4SLinus Torvalds if (sk->sk_state != TCP_CLOSE) { 25801da177e4SLinus Torvalds err = -EINVAL; 25811da177e4SLinus Torvalds break; 25821da177e4SLinus Torvalds } 25831da177e4SLinus Torvalds tp->window_clamp = 0; 25841da177e4SLinus Torvalds } else 25851da177e4SLinus Torvalds tp->window_clamp = val < SOCK_MIN_RCVBUF / 2 ? 25861da177e4SLinus Torvalds SOCK_MIN_RCVBUF / 2 : val; 25871da177e4SLinus Torvalds break; 25881da177e4SLinus Torvalds 25891da177e4SLinus Torvalds case TCP_QUICKACK: 25901da177e4SLinus Torvalds if (!val) { 2591463c84b9SArnaldo Carvalho de Melo icsk->icsk_ack.pingpong = 1; 25921da177e4SLinus Torvalds } else { 2593463c84b9SArnaldo Carvalho de Melo icsk->icsk_ack.pingpong = 0; 25941da177e4SLinus Torvalds if ((1 << sk->sk_state) & 25951da177e4SLinus Torvalds (TCPF_ESTABLISHED | TCPF_CLOSE_WAIT) && 2596463c84b9SArnaldo Carvalho de Melo inet_csk_ack_scheduled(sk)) { 2597463c84b9SArnaldo Carvalho de Melo icsk->icsk_ack.pending |= ICSK_ACK_PUSHED; 25980e4b4992SChris Leech tcp_cleanup_rbuf(sk, 1); 25991da177e4SLinus Torvalds if (!(val & 1)) 2600463c84b9SArnaldo Carvalho de Melo icsk->icsk_ack.pingpong = 1; 26011da177e4SLinus Torvalds } 26021da177e4SLinus Torvalds } 26031da177e4SLinus Torvalds break; 26041da177e4SLinus Torvalds 2605cfb6eeb4SYOSHIFUJI Hideaki #ifdef CONFIG_TCP_MD5SIG 2606cfb6eeb4SYOSHIFUJI Hideaki case TCP_MD5SIG: 2607cfb6eeb4SYOSHIFUJI Hideaki /* Read the IP->Key mappings from userspace */ 2608cfb6eeb4SYOSHIFUJI Hideaki err = tp->af_specific->md5_parse(sk, optval, optlen); 2609cfb6eeb4SYOSHIFUJI Hideaki break; 2610cfb6eeb4SYOSHIFUJI Hideaki #endif 2611dca43c75SJerry Chu case TCP_USER_TIMEOUT: 2612dca43c75SJerry Chu /* Cap the max timeout in ms TCP will retry/retrans 2613dca43c75SJerry Chu * before giving up and aborting (ETIMEDOUT) a connection. 2614dca43c75SJerry Chu */ 261542493570SHangbin Liu if (val < 0) 261642493570SHangbin Liu err = -EINVAL; 261742493570SHangbin Liu else 2618dca43c75SJerry Chu icsk->icsk_user_timeout = msecs_to_jiffies(val); 2619dca43c75SJerry Chu break; 26208336886fSJerry Chu 26218336886fSJerry Chu case TCP_FASTOPEN: 26228336886fSJerry Chu if (val >= 0 && ((1 << sk->sk_state) & (TCPF_CLOSE | 26238336886fSJerry Chu TCPF_LISTEN))) 26248336886fSJerry Chu err = fastopen_init_queue(sk, val); 26258336886fSJerry Chu else 26268336886fSJerry Chu err = -EINVAL; 26278336886fSJerry Chu break; 262893be6ce0SAndrey Vagin case TCP_TIMESTAMP: 262993be6ce0SAndrey Vagin if (!tp->repair) 263093be6ce0SAndrey Vagin err = -EPERM; 263193be6ce0SAndrey Vagin else 263293be6ce0SAndrey Vagin tp->tsoffset = val - tcp_time_stamp; 263393be6ce0SAndrey Vagin break; 26341da177e4SLinus Torvalds default: 26351da177e4SLinus Torvalds err = -ENOPROTOOPT; 26361da177e4SLinus Torvalds break; 26373ff50b79SStephen Hemminger } 26383ff50b79SStephen Hemminger 26391da177e4SLinus Torvalds release_sock(sk); 26401da177e4SLinus Torvalds return err; 26411da177e4SLinus Torvalds } 26421da177e4SLinus Torvalds 26433fdadf7dSDmitry Mishin int tcp_setsockopt(struct sock *sk, int level, int optname, char __user *optval, 2644b7058842SDavid S. Miller unsigned int optlen) 26453fdadf7dSDmitry Mishin { 2646cf533ea5SEric Dumazet const struct inet_connection_sock *icsk = inet_csk(sk); 26473fdadf7dSDmitry Mishin 26483fdadf7dSDmitry Mishin if (level != SOL_TCP) 26493fdadf7dSDmitry Mishin return icsk->icsk_af_ops->setsockopt(sk, level, optname, 26503fdadf7dSDmitry Mishin optval, optlen); 26513fdadf7dSDmitry Mishin return do_tcp_setsockopt(sk, level, optname, optval, optlen); 26523fdadf7dSDmitry Mishin } 26534bc2f18bSEric Dumazet EXPORT_SYMBOL(tcp_setsockopt); 26543fdadf7dSDmitry Mishin 26553fdadf7dSDmitry Mishin #ifdef CONFIG_COMPAT 2656543d9cfeSArnaldo Carvalho de Melo int compat_tcp_setsockopt(struct sock *sk, int level, int optname, 2657b7058842SDavid S. Miller char __user *optval, unsigned int optlen) 26583fdadf7dSDmitry Mishin { 2659dec73ff0SArnaldo Carvalho de Melo if (level != SOL_TCP) 2660dec73ff0SArnaldo Carvalho de Melo return inet_csk_compat_setsockopt(sk, level, optname, 2661dec73ff0SArnaldo Carvalho de Melo optval, optlen); 26623fdadf7dSDmitry Mishin return do_tcp_setsockopt(sk, level, optname, optval, optlen); 26633fdadf7dSDmitry Mishin } 2664543d9cfeSArnaldo Carvalho de Melo EXPORT_SYMBOL(compat_tcp_setsockopt); 26653fdadf7dSDmitry Mishin #endif 26663fdadf7dSDmitry Mishin 26671da177e4SLinus Torvalds /* Return information about state of tcp endpoint in API format. */ 2668cf533ea5SEric Dumazet void tcp_get_info(const struct sock *sk, struct tcp_info *info) 26691da177e4SLinus Torvalds { 2670cf533ea5SEric Dumazet const struct tcp_sock *tp = tcp_sk(sk); 2671463c84b9SArnaldo Carvalho de Melo const struct inet_connection_sock *icsk = inet_csk(sk); 26721da177e4SLinus Torvalds u32 now = tcp_time_stamp; 26731da177e4SLinus Torvalds 26741da177e4SLinus Torvalds memset(info, 0, sizeof(*info)); 26751da177e4SLinus Torvalds 26761da177e4SLinus Torvalds info->tcpi_state = sk->sk_state; 26776687e988SArnaldo Carvalho de Melo info->tcpi_ca_state = icsk->icsk_ca_state; 2678463c84b9SArnaldo Carvalho de Melo info->tcpi_retransmits = icsk->icsk_retransmits; 26796687e988SArnaldo Carvalho de Melo info->tcpi_probes = icsk->icsk_probes_out; 2680463c84b9SArnaldo Carvalho de Melo info->tcpi_backoff = icsk->icsk_backoff; 26811da177e4SLinus Torvalds 26821da177e4SLinus Torvalds if (tp->rx_opt.tstamp_ok) 26831da177e4SLinus Torvalds info->tcpi_options |= TCPI_OPT_TIMESTAMPS; 2684e60402d0SIlpo Järvinen if (tcp_is_sack(tp)) 26851da177e4SLinus Torvalds info->tcpi_options |= TCPI_OPT_SACK; 26861da177e4SLinus Torvalds if (tp->rx_opt.wscale_ok) { 26871da177e4SLinus Torvalds info->tcpi_options |= TCPI_OPT_WSCALE; 26881da177e4SLinus Torvalds info->tcpi_snd_wscale = tp->rx_opt.snd_wscale; 26891da177e4SLinus Torvalds info->tcpi_rcv_wscale = tp->rx_opt.rcv_wscale; 26901da177e4SLinus Torvalds } 26911da177e4SLinus Torvalds 26921da177e4SLinus Torvalds if (tp->ecn_flags & TCP_ECN_OK) 26931da177e4SLinus Torvalds info->tcpi_options |= TCPI_OPT_ECN; 2694b5c5693bSEric Dumazet if (tp->ecn_flags & TCP_ECN_SEEN) 2695b5c5693bSEric Dumazet info->tcpi_options |= TCPI_OPT_ECN_SEEN; 26966f73601eSYuchung Cheng if (tp->syn_data_acked) 26976f73601eSYuchung Cheng info->tcpi_options |= TCPI_OPT_SYN_DATA; 26981da177e4SLinus Torvalds 2699463c84b9SArnaldo Carvalho de Melo info->tcpi_rto = jiffies_to_usecs(icsk->icsk_rto); 2700463c84b9SArnaldo Carvalho de Melo info->tcpi_ato = jiffies_to_usecs(icsk->icsk_ack.ato); 2701c1b4a7e6SDavid S. Miller info->tcpi_snd_mss = tp->mss_cache; 2702463c84b9SArnaldo Carvalho de Melo info->tcpi_rcv_mss = icsk->icsk_ack.rcv_mss; 27031da177e4SLinus Torvalds 27045ee3afbaSRick Jones if (sk->sk_state == TCP_LISTEN) { 27055ee3afbaSRick Jones info->tcpi_unacked = sk->sk_ack_backlog; 27065ee3afbaSRick Jones info->tcpi_sacked = sk->sk_max_ack_backlog; 27075ee3afbaSRick Jones } else { 27081da177e4SLinus Torvalds info->tcpi_unacked = tp->packets_out; 27091da177e4SLinus Torvalds info->tcpi_sacked = tp->sacked_out; 27105ee3afbaSRick Jones } 27111da177e4SLinus Torvalds info->tcpi_lost = tp->lost_out; 27121da177e4SLinus Torvalds info->tcpi_retrans = tp->retrans_out; 27131da177e4SLinus Torvalds info->tcpi_fackets = tp->fackets_out; 27141da177e4SLinus Torvalds 27151da177e4SLinus Torvalds info->tcpi_last_data_sent = jiffies_to_msecs(now - tp->lsndtime); 2716463c84b9SArnaldo Carvalho de Melo info->tcpi_last_data_recv = jiffies_to_msecs(now - icsk->icsk_ack.lrcvtime); 27171da177e4SLinus Torvalds info->tcpi_last_ack_recv = jiffies_to_msecs(now - tp->rcv_tstamp); 27181da177e4SLinus Torvalds 2719d83d8461SArnaldo Carvalho de Melo info->tcpi_pmtu = icsk->icsk_pmtu_cookie; 27201da177e4SLinus Torvalds info->tcpi_rcv_ssthresh = tp->rcv_ssthresh; 27211da177e4SLinus Torvalds info->tcpi_rtt = jiffies_to_usecs(tp->srtt)>>3; 27221da177e4SLinus Torvalds info->tcpi_rttvar = jiffies_to_usecs(tp->mdev)>>2; 27231da177e4SLinus Torvalds info->tcpi_snd_ssthresh = tp->snd_ssthresh; 27241da177e4SLinus Torvalds info->tcpi_snd_cwnd = tp->snd_cwnd; 27251da177e4SLinus Torvalds info->tcpi_advmss = tp->advmss; 27261da177e4SLinus Torvalds info->tcpi_reordering = tp->reordering; 27271da177e4SLinus Torvalds 27281da177e4SLinus Torvalds info->tcpi_rcv_rtt = jiffies_to_usecs(tp->rcv_rtt_est.rtt)>>3; 27291da177e4SLinus Torvalds info->tcpi_rcv_space = tp->rcvq_space.space; 27301da177e4SLinus Torvalds 27311da177e4SLinus Torvalds info->tcpi_total_retrans = tp->total_retrans; 27321da177e4SLinus Torvalds } 27331da177e4SLinus Torvalds EXPORT_SYMBOL_GPL(tcp_get_info); 27341da177e4SLinus Torvalds 27353fdadf7dSDmitry Mishin static int do_tcp_getsockopt(struct sock *sk, int level, 27363fdadf7dSDmitry Mishin int optname, char __user *optval, int __user *optlen) 27371da177e4SLinus Torvalds { 2738295f7324SArnaldo Carvalho de Melo struct inet_connection_sock *icsk = inet_csk(sk); 27391da177e4SLinus Torvalds struct tcp_sock *tp = tcp_sk(sk); 27401da177e4SLinus Torvalds int val, len; 27411da177e4SLinus Torvalds 27421da177e4SLinus Torvalds if (get_user(len, optlen)) 27431da177e4SLinus Torvalds return -EFAULT; 27441da177e4SLinus Torvalds 27451da177e4SLinus Torvalds len = min_t(unsigned int, len, sizeof(int)); 27461da177e4SLinus Torvalds 27471da177e4SLinus Torvalds if (len < 0) 27481da177e4SLinus Torvalds return -EINVAL; 27491da177e4SLinus Torvalds 27501da177e4SLinus Torvalds switch (optname) { 27511da177e4SLinus Torvalds case TCP_MAXSEG: 2752c1b4a7e6SDavid S. Miller val = tp->mss_cache; 27531da177e4SLinus Torvalds if (!val && ((1 << sk->sk_state) & (TCPF_CLOSE | TCPF_LISTEN))) 27541da177e4SLinus Torvalds val = tp->rx_opt.user_mss; 27555e6a3ce6SPavel Emelyanov if (tp->repair) 27565e6a3ce6SPavel Emelyanov val = tp->rx_opt.mss_clamp; 27571da177e4SLinus Torvalds break; 27581da177e4SLinus Torvalds case TCP_NODELAY: 27591da177e4SLinus Torvalds val = !!(tp->nonagle&TCP_NAGLE_OFF); 27601da177e4SLinus Torvalds break; 27611da177e4SLinus Torvalds case TCP_CORK: 27621da177e4SLinus Torvalds val = !!(tp->nonagle&TCP_NAGLE_CORK); 27631da177e4SLinus Torvalds break; 27641da177e4SLinus Torvalds case TCP_KEEPIDLE: 2765df19a626SEric Dumazet val = keepalive_time_when(tp) / HZ; 27661da177e4SLinus Torvalds break; 27671da177e4SLinus Torvalds case TCP_KEEPINTVL: 2768df19a626SEric Dumazet val = keepalive_intvl_when(tp) / HZ; 27691da177e4SLinus Torvalds break; 27701da177e4SLinus Torvalds case TCP_KEEPCNT: 2771df19a626SEric Dumazet val = keepalive_probes(tp); 27721da177e4SLinus Torvalds break; 27731da177e4SLinus Torvalds case TCP_SYNCNT: 2774295f7324SArnaldo Carvalho de Melo val = icsk->icsk_syn_retries ? : sysctl_tcp_syn_retries; 27751da177e4SLinus Torvalds break; 27761da177e4SLinus Torvalds case TCP_LINGER2: 27771da177e4SLinus Torvalds val = tp->linger2; 27781da177e4SLinus Torvalds if (val >= 0) 27791da177e4SLinus Torvalds val = (val ? : sysctl_tcp_fin_timeout) / HZ; 27801da177e4SLinus Torvalds break; 27811da177e4SLinus Torvalds case TCP_DEFER_ACCEPT: 2782b103cf34SJulian Anastasov val = retrans_to_secs(icsk->icsk_accept_queue.rskq_defer_accept, 2783b103cf34SJulian Anastasov TCP_TIMEOUT_INIT / HZ, TCP_RTO_MAX / HZ); 27841da177e4SLinus Torvalds break; 27851da177e4SLinus Torvalds case TCP_WINDOW_CLAMP: 27861da177e4SLinus Torvalds val = tp->window_clamp; 27871da177e4SLinus Torvalds break; 27881da177e4SLinus Torvalds case TCP_INFO: { 27891da177e4SLinus Torvalds struct tcp_info info; 27901da177e4SLinus Torvalds 27911da177e4SLinus Torvalds if (get_user(len, optlen)) 27921da177e4SLinus Torvalds return -EFAULT; 27931da177e4SLinus Torvalds 27941da177e4SLinus Torvalds tcp_get_info(sk, &info); 27951da177e4SLinus Torvalds 27961da177e4SLinus Torvalds len = min_t(unsigned int, len, sizeof(info)); 27971da177e4SLinus Torvalds if (put_user(len, optlen)) 27981da177e4SLinus Torvalds return -EFAULT; 27991da177e4SLinus Torvalds if (copy_to_user(optval, &info, len)) 28001da177e4SLinus Torvalds return -EFAULT; 28011da177e4SLinus Torvalds return 0; 28021da177e4SLinus Torvalds } 28031da177e4SLinus Torvalds case TCP_QUICKACK: 2804295f7324SArnaldo Carvalho de Melo val = !icsk->icsk_ack.pingpong; 28051da177e4SLinus Torvalds break; 28065f8ef48dSStephen Hemminger 28075f8ef48dSStephen Hemminger case TCP_CONGESTION: 28085f8ef48dSStephen Hemminger if (get_user(len, optlen)) 28095f8ef48dSStephen Hemminger return -EFAULT; 28105f8ef48dSStephen Hemminger len = min_t(unsigned int, len, TCP_CA_NAME_MAX); 28115f8ef48dSStephen Hemminger if (put_user(len, optlen)) 28125f8ef48dSStephen Hemminger return -EFAULT; 28136687e988SArnaldo Carvalho de Melo if (copy_to_user(optval, icsk->icsk_ca_ops->name, len)) 28145f8ef48dSStephen Hemminger return -EFAULT; 28155f8ef48dSStephen Hemminger return 0; 2816e56fb50fSWilliam Allen Simpson 28173c0fef0bSJosh Hunt case TCP_THIN_LINEAR_TIMEOUTS: 28183c0fef0bSJosh Hunt val = tp->thin_lto; 28193c0fef0bSJosh Hunt break; 28203c0fef0bSJosh Hunt case TCP_THIN_DUPACK: 28213c0fef0bSJosh Hunt val = tp->thin_dupack; 28223c0fef0bSJosh Hunt break; 2823dca43c75SJerry Chu 2824ee995283SPavel Emelyanov case TCP_REPAIR: 2825ee995283SPavel Emelyanov val = tp->repair; 2826ee995283SPavel Emelyanov break; 2827ee995283SPavel Emelyanov 2828ee995283SPavel Emelyanov case TCP_REPAIR_QUEUE: 2829ee995283SPavel Emelyanov if (tp->repair) 2830ee995283SPavel Emelyanov val = tp->repair_queue; 2831ee995283SPavel Emelyanov else 2832ee995283SPavel Emelyanov return -EINVAL; 2833ee995283SPavel Emelyanov break; 2834ee995283SPavel Emelyanov 2835ee995283SPavel Emelyanov case TCP_QUEUE_SEQ: 2836ee995283SPavel Emelyanov if (tp->repair_queue == TCP_SEND_QUEUE) 2837ee995283SPavel Emelyanov val = tp->write_seq; 2838ee995283SPavel Emelyanov else if (tp->repair_queue == TCP_RECV_QUEUE) 2839ee995283SPavel Emelyanov val = tp->rcv_nxt; 2840ee995283SPavel Emelyanov else 2841ee995283SPavel Emelyanov return -EINVAL; 2842ee995283SPavel Emelyanov break; 2843ee995283SPavel Emelyanov 2844dca43c75SJerry Chu case TCP_USER_TIMEOUT: 2845dca43c75SJerry Chu val = jiffies_to_msecs(icsk->icsk_user_timeout); 2846dca43c75SJerry Chu break; 284793be6ce0SAndrey Vagin case TCP_TIMESTAMP: 284893be6ce0SAndrey Vagin val = tcp_time_stamp + tp->tsoffset; 284993be6ce0SAndrey Vagin break; 28501da177e4SLinus Torvalds default: 28511da177e4SLinus Torvalds return -ENOPROTOOPT; 28523ff50b79SStephen Hemminger } 28531da177e4SLinus Torvalds 28541da177e4SLinus Torvalds if (put_user(len, optlen)) 28551da177e4SLinus Torvalds return -EFAULT; 28561da177e4SLinus Torvalds if (copy_to_user(optval, &val, len)) 28571da177e4SLinus Torvalds return -EFAULT; 28581da177e4SLinus Torvalds return 0; 28591da177e4SLinus Torvalds } 28601da177e4SLinus Torvalds 28613fdadf7dSDmitry Mishin int tcp_getsockopt(struct sock *sk, int level, int optname, char __user *optval, 28623fdadf7dSDmitry Mishin int __user *optlen) 28633fdadf7dSDmitry Mishin { 28643fdadf7dSDmitry Mishin struct inet_connection_sock *icsk = inet_csk(sk); 28653fdadf7dSDmitry Mishin 28663fdadf7dSDmitry Mishin if (level != SOL_TCP) 28673fdadf7dSDmitry Mishin return icsk->icsk_af_ops->getsockopt(sk, level, optname, 28683fdadf7dSDmitry Mishin optval, optlen); 28693fdadf7dSDmitry Mishin return do_tcp_getsockopt(sk, level, optname, optval, optlen); 28703fdadf7dSDmitry Mishin } 28714bc2f18bSEric Dumazet EXPORT_SYMBOL(tcp_getsockopt); 28723fdadf7dSDmitry Mishin 28733fdadf7dSDmitry Mishin #ifdef CONFIG_COMPAT 2874543d9cfeSArnaldo Carvalho de Melo int compat_tcp_getsockopt(struct sock *sk, int level, int optname, 2875543d9cfeSArnaldo Carvalho de Melo char __user *optval, int __user *optlen) 28763fdadf7dSDmitry Mishin { 2877dec73ff0SArnaldo Carvalho de Melo if (level != SOL_TCP) 2878dec73ff0SArnaldo Carvalho de Melo return inet_csk_compat_getsockopt(sk, level, optname, 2879dec73ff0SArnaldo Carvalho de Melo optval, optlen); 28803fdadf7dSDmitry Mishin return do_tcp_getsockopt(sk, level, optname, optval, optlen); 28813fdadf7dSDmitry Mishin } 2882543d9cfeSArnaldo Carvalho de Melo EXPORT_SYMBOL(compat_tcp_getsockopt); 28833fdadf7dSDmitry Mishin #endif 28841da177e4SLinus Torvalds 2885cfb6eeb4SYOSHIFUJI Hideaki #ifdef CONFIG_TCP_MD5SIG 288671cea17eSEric Dumazet static struct tcp_md5sig_pool __percpu *tcp_md5sig_pool __read_mostly; 288771cea17eSEric Dumazet static DEFINE_MUTEX(tcp_md5sig_mutex); 2888cfb6eeb4SYOSHIFUJI Hideaki 2889765cf997SEric Dumazet static void __tcp_free_md5sig_pool(struct tcp_md5sig_pool __percpu *pool) 2890cfb6eeb4SYOSHIFUJI Hideaki { 2891cfb6eeb4SYOSHIFUJI Hideaki int cpu; 2892765cf997SEric Dumazet 2893cfb6eeb4SYOSHIFUJI Hideaki for_each_possible_cpu(cpu) { 2894765cf997SEric Dumazet struct tcp_md5sig_pool *p = per_cpu_ptr(pool, cpu); 2895765cf997SEric Dumazet 2896cfb6eeb4SYOSHIFUJI Hideaki if (p->md5_desc.tfm) 2897cfb6eeb4SYOSHIFUJI Hideaki crypto_free_hash(p->md5_desc.tfm); 2898cfb6eeb4SYOSHIFUJI Hideaki } 2899cfb6eeb4SYOSHIFUJI Hideaki free_percpu(pool); 2900cfb6eeb4SYOSHIFUJI Hideaki } 2901cfb6eeb4SYOSHIFUJI Hideaki 290271cea17eSEric Dumazet static void __tcp_alloc_md5sig_pool(void) 2903cfb6eeb4SYOSHIFUJI Hideaki { 2904cfb6eeb4SYOSHIFUJI Hideaki int cpu; 2905765cf997SEric Dumazet struct tcp_md5sig_pool __percpu *pool; 2906cfb6eeb4SYOSHIFUJI Hideaki 2907765cf997SEric Dumazet pool = alloc_percpu(struct tcp_md5sig_pool); 2908cfb6eeb4SYOSHIFUJI Hideaki if (!pool) 290971cea17eSEric Dumazet return; 2910cfb6eeb4SYOSHIFUJI Hideaki 2911cfb6eeb4SYOSHIFUJI Hideaki for_each_possible_cpu(cpu) { 2912cfb6eeb4SYOSHIFUJI Hideaki struct crypto_hash *hash; 2913cfb6eeb4SYOSHIFUJI Hideaki 2914cfb6eeb4SYOSHIFUJI Hideaki hash = crypto_alloc_hash("md5", 0, CRYPTO_ALG_ASYNC); 291550c3a487SYOSHIFUJI Hideaki / 吉藤英明 if (IS_ERR_OR_NULL(hash)) 2916cfb6eeb4SYOSHIFUJI Hideaki goto out_free; 2917cfb6eeb4SYOSHIFUJI Hideaki 2918765cf997SEric Dumazet per_cpu_ptr(pool, cpu)->md5_desc.tfm = hash; 2919cfb6eeb4SYOSHIFUJI Hideaki } 292071cea17eSEric Dumazet /* before setting tcp_md5sig_pool, we must commit all writes 292171cea17eSEric Dumazet * to memory. See ACCESS_ONCE() in tcp_get_md5sig_pool() 292271cea17eSEric Dumazet */ 292371cea17eSEric Dumazet smp_wmb(); 292471cea17eSEric Dumazet tcp_md5sig_pool = pool; 292571cea17eSEric Dumazet return; 2926cfb6eeb4SYOSHIFUJI Hideaki out_free: 2927cfb6eeb4SYOSHIFUJI Hideaki __tcp_free_md5sig_pool(pool); 2928cfb6eeb4SYOSHIFUJI Hideaki } 2929cfb6eeb4SYOSHIFUJI Hideaki 293071cea17eSEric Dumazet bool tcp_alloc_md5sig_pool(void) 2931cfb6eeb4SYOSHIFUJI Hideaki { 293271cea17eSEric Dumazet if (unlikely(!tcp_md5sig_pool)) { 293371cea17eSEric Dumazet mutex_lock(&tcp_md5sig_mutex); 2934cfb6eeb4SYOSHIFUJI Hideaki 293571cea17eSEric Dumazet if (!tcp_md5sig_pool) 293671cea17eSEric Dumazet __tcp_alloc_md5sig_pool(); 2937cfb6eeb4SYOSHIFUJI Hideaki 293871cea17eSEric Dumazet mutex_unlock(&tcp_md5sig_mutex); 2939cfb6eeb4SYOSHIFUJI Hideaki } 294071cea17eSEric Dumazet return tcp_md5sig_pool != NULL; 2941cfb6eeb4SYOSHIFUJI Hideaki } 2942cfb6eeb4SYOSHIFUJI Hideaki EXPORT_SYMBOL(tcp_alloc_md5sig_pool); 2943cfb6eeb4SYOSHIFUJI Hideaki 294435790c04SEric Dumazet 294535790c04SEric Dumazet /** 294635790c04SEric Dumazet * tcp_get_md5sig_pool - get md5sig_pool for this user 294735790c04SEric Dumazet * 294835790c04SEric Dumazet * We use percpu structure, so if we succeed, we exit with preemption 294935790c04SEric Dumazet * and BH disabled, to make sure another thread or softirq handling 295035790c04SEric Dumazet * wont try to get same context. 295135790c04SEric Dumazet */ 295235790c04SEric Dumazet struct tcp_md5sig_pool *tcp_get_md5sig_pool(void) 2953cfb6eeb4SYOSHIFUJI Hideaki { 2954765cf997SEric Dumazet struct tcp_md5sig_pool __percpu *p; 295535790c04SEric Dumazet 295635790c04SEric Dumazet local_bh_disable(); 295771cea17eSEric Dumazet p = ACCESS_ONCE(tcp_md5sig_pool); 2958cfb6eeb4SYOSHIFUJI Hideaki if (p) 295971cea17eSEric Dumazet return __this_cpu_ptr(p); 296035790c04SEric Dumazet 296135790c04SEric Dumazet local_bh_enable(); 296235790c04SEric Dumazet return NULL; 2963cfb6eeb4SYOSHIFUJI Hideaki } 296435790c04SEric Dumazet EXPORT_SYMBOL(tcp_get_md5sig_pool); 2965cfb6eeb4SYOSHIFUJI Hideaki 296649a72dfbSAdam Langley int tcp_md5_hash_header(struct tcp_md5sig_pool *hp, 2967ca35a0efSEric Dumazet const struct tcphdr *th) 296849a72dfbSAdam Langley { 296949a72dfbSAdam Langley struct scatterlist sg; 2970ca35a0efSEric Dumazet struct tcphdr hdr; 297149a72dfbSAdam Langley int err; 297249a72dfbSAdam Langley 2973ca35a0efSEric Dumazet /* We are not allowed to change tcphdr, make a local copy */ 2974ca35a0efSEric Dumazet memcpy(&hdr, th, sizeof(hdr)); 2975ca35a0efSEric Dumazet hdr.check = 0; 2976ca35a0efSEric Dumazet 297749a72dfbSAdam Langley /* options aren't included in the hash */ 2978ca35a0efSEric Dumazet sg_init_one(&sg, &hdr, sizeof(hdr)); 2979ca35a0efSEric Dumazet err = crypto_hash_update(&hp->md5_desc, &sg, sizeof(hdr)); 298049a72dfbSAdam Langley return err; 298149a72dfbSAdam Langley } 298249a72dfbSAdam Langley EXPORT_SYMBOL(tcp_md5_hash_header); 298349a72dfbSAdam Langley 298449a72dfbSAdam Langley int tcp_md5_hash_skb_data(struct tcp_md5sig_pool *hp, 2985cf533ea5SEric Dumazet const struct sk_buff *skb, unsigned int header_len) 298649a72dfbSAdam Langley { 298749a72dfbSAdam Langley struct scatterlist sg; 298849a72dfbSAdam Langley const struct tcphdr *tp = tcp_hdr(skb); 298949a72dfbSAdam Langley struct hash_desc *desc = &hp->md5_desc; 299095c96174SEric Dumazet unsigned int i; 299195c96174SEric Dumazet const unsigned int head_data_len = skb_headlen(skb) > header_len ? 299249a72dfbSAdam Langley skb_headlen(skb) - header_len : 0; 299349a72dfbSAdam Langley const struct skb_shared_info *shi = skb_shinfo(skb); 2994d7fd1b57SEric Dumazet struct sk_buff *frag_iter; 299549a72dfbSAdam Langley 299649a72dfbSAdam Langley sg_init_table(&sg, 1); 299749a72dfbSAdam Langley 299849a72dfbSAdam Langley sg_set_buf(&sg, ((u8 *) tp) + header_len, head_data_len); 299949a72dfbSAdam Langley if (crypto_hash_update(desc, &sg, head_data_len)) 300049a72dfbSAdam Langley return 1; 300149a72dfbSAdam Langley 300249a72dfbSAdam Langley for (i = 0; i < shi->nr_frags; ++i) { 300349a72dfbSAdam Langley const struct skb_frag_struct *f = &shi->frags[i]; 300454d27fcbSEric Dumazet unsigned int offset = f->page_offset; 300554d27fcbSEric Dumazet struct page *page = skb_frag_page(f) + (offset >> PAGE_SHIFT); 300654d27fcbSEric Dumazet 300754d27fcbSEric Dumazet sg_set_page(&sg, page, skb_frag_size(f), 300854d27fcbSEric Dumazet offset_in_page(offset)); 30099e903e08SEric Dumazet if (crypto_hash_update(desc, &sg, skb_frag_size(f))) 301049a72dfbSAdam Langley return 1; 301149a72dfbSAdam Langley } 301249a72dfbSAdam Langley 3013d7fd1b57SEric Dumazet skb_walk_frags(skb, frag_iter) 3014d7fd1b57SEric Dumazet if (tcp_md5_hash_skb_data(hp, frag_iter, 0)) 3015d7fd1b57SEric Dumazet return 1; 3016d7fd1b57SEric Dumazet 301749a72dfbSAdam Langley return 0; 301849a72dfbSAdam Langley } 301949a72dfbSAdam Langley EXPORT_SYMBOL(tcp_md5_hash_skb_data); 302049a72dfbSAdam Langley 3021cf533ea5SEric Dumazet int tcp_md5_hash_key(struct tcp_md5sig_pool *hp, const struct tcp_md5sig_key *key) 302249a72dfbSAdam Langley { 302349a72dfbSAdam Langley struct scatterlist sg; 302449a72dfbSAdam Langley 302549a72dfbSAdam Langley sg_init_one(&sg, key->key, key->keylen); 302649a72dfbSAdam Langley return crypto_hash_update(&hp->md5_desc, &sg, key->keylen); 302749a72dfbSAdam Langley } 302849a72dfbSAdam Langley EXPORT_SYMBOL(tcp_md5_hash_key); 302949a72dfbSAdam Langley 3030cfb6eeb4SYOSHIFUJI Hideaki #endif 3031cfb6eeb4SYOSHIFUJI Hideaki 30324ac02babSAndi Kleen void tcp_done(struct sock *sk) 30334ac02babSAndi Kleen { 30348336886fSJerry Chu struct request_sock *req = tcp_sk(sk)->fastopen_rsk; 30358336886fSJerry Chu 30364ac02babSAndi Kleen if (sk->sk_state == TCP_SYN_SENT || sk->sk_state == TCP_SYN_RECV) 303763231bddSPavel Emelyanov TCP_INC_STATS_BH(sock_net(sk), TCP_MIB_ATTEMPTFAILS); 30384ac02babSAndi Kleen 30394ac02babSAndi Kleen tcp_set_state(sk, TCP_CLOSE); 30404ac02babSAndi Kleen tcp_clear_xmit_timers(sk); 30418336886fSJerry Chu if (req != NULL) 30428336886fSJerry Chu reqsk_fastopen_remove(sk, req, false); 30434ac02babSAndi Kleen 30444ac02babSAndi Kleen sk->sk_shutdown = SHUTDOWN_MASK; 30454ac02babSAndi Kleen 30464ac02babSAndi Kleen if (!sock_flag(sk, SOCK_DEAD)) 30474ac02babSAndi Kleen sk->sk_state_change(sk); 30484ac02babSAndi Kleen else 30494ac02babSAndi Kleen inet_csk_destroy_sock(sk); 30504ac02babSAndi Kleen } 30514ac02babSAndi Kleen EXPORT_SYMBOL_GPL(tcp_done); 30524ac02babSAndi Kleen 30535f8ef48dSStephen Hemminger extern struct tcp_congestion_ops tcp_reno; 30541da177e4SLinus Torvalds 30551da177e4SLinus Torvalds static __initdata unsigned long thash_entries; 30561da177e4SLinus Torvalds static int __init set_thash_entries(char *str) 30571da177e4SLinus Torvalds { 3058413c27d8SEldad Zack ssize_t ret; 3059413c27d8SEldad Zack 30601da177e4SLinus Torvalds if (!str) 30611da177e4SLinus Torvalds return 0; 3062413c27d8SEldad Zack 3063413c27d8SEldad Zack ret = kstrtoul(str, 0, &thash_entries); 3064413c27d8SEldad Zack if (ret) 3065413c27d8SEldad Zack return 0; 3066413c27d8SEldad Zack 30671da177e4SLinus Torvalds return 1; 30681da177e4SLinus Torvalds } 30691da177e4SLinus Torvalds __setup("thash_entries=", set_thash_entries); 30701da177e4SLinus Torvalds 30714acb4190SGlauber Costa void tcp_init_mem(struct net *net) 30724acb4190SGlauber Costa { 30734acb4190SGlauber Costa unsigned long limit = nr_free_buffer_pages() / 8; 30744acb4190SGlauber Costa limit = max(limit, 128UL); 30754acb4190SGlauber Costa net->ipv4.sysctl_tcp_mem[0] = limit / 4 * 3; 30764acb4190SGlauber Costa net->ipv4.sysctl_tcp_mem[1] = limit; 30774acb4190SGlauber Costa net->ipv4.sysctl_tcp_mem[2] = net->ipv4.sysctl_tcp_mem[0] * 2; 30784acb4190SGlauber Costa } 30794acb4190SGlauber Costa 30801da177e4SLinus Torvalds void __init tcp_init(void) 30811da177e4SLinus Torvalds { 30821da177e4SLinus Torvalds struct sk_buff *skb = NULL; 3083f03d78dbSEric Dumazet unsigned long limit; 3084b49960a0SEric Dumazet int max_rshare, max_wshare, cnt; 3085074b8517SDimitri Sivanich unsigned int i; 30861da177e4SLinus Torvalds 30871f9e636eSPavel Emelyanov BUILD_BUG_ON(sizeof(struct tcp_skb_cb) > sizeof(skb->cb)); 30881da177e4SLinus Torvalds 30891748376bSEric Dumazet percpu_counter_init(&tcp_sockets_allocated, 0); 3090dd24c001SEric Dumazet percpu_counter_init(&tcp_orphan_count, 0); 30916e04e021SArnaldo Carvalho de Melo tcp_hashinfo.bind_bucket_cachep = 30926e04e021SArnaldo Carvalho de Melo kmem_cache_create("tcp_bind_bucket", 30936e04e021SArnaldo Carvalho de Melo sizeof(struct inet_bind_bucket), 0, 309420c2df83SPaul Mundt SLAB_HWCACHE_ALIGN|SLAB_PANIC, NULL); 30951da177e4SLinus Torvalds 30961da177e4SLinus Torvalds /* Size and allocate the main established and bind bucket 30971da177e4SLinus Torvalds * hash tables. 30981da177e4SLinus Torvalds * 30991da177e4SLinus Torvalds * The methodology is similar to that of the buffer cache. 31001da177e4SLinus Torvalds */ 31016e04e021SArnaldo Carvalho de Melo tcp_hashinfo.ehash = 31021da177e4SLinus Torvalds alloc_large_system_hash("TCP established", 31030f7ff927SArnaldo Carvalho de Melo sizeof(struct inet_ehash_bucket), 31041da177e4SLinus Torvalds thash_entries, 3105fd90b29dSEric Dumazet 17, /* one slot per 128 KB of memory */ 31069e950efaSJohn Heffner 0, 31071da177e4SLinus Torvalds NULL, 3108f373b53bSEric Dumazet &tcp_hashinfo.ehash_mask, 310931fe62b9STim Bird 0, 31100ccfe618SJean Delvare thash_entries ? 0 : 512 * 1024); 3111f373b53bSEric Dumazet for (i = 0; i <= tcp_hashinfo.ehash_mask; i++) { 31123ab5aee7SEric Dumazet INIT_HLIST_NULLS_HEAD(&tcp_hashinfo.ehash[i].chain, i); 31133ab5aee7SEric Dumazet INIT_HLIST_NULLS_HEAD(&tcp_hashinfo.ehash[i].twchain, i); 31141da177e4SLinus Torvalds } 3115230140cfSEric Dumazet if (inet_ehash_locks_alloc(&tcp_hashinfo)) 3116230140cfSEric Dumazet panic("TCP: failed to alloc ehash_locks"); 31176e04e021SArnaldo Carvalho de Melo tcp_hashinfo.bhash = 31181da177e4SLinus Torvalds alloc_large_system_hash("TCP bind", 31190f7ff927SArnaldo Carvalho de Melo sizeof(struct inet_bind_hashbucket), 3120f373b53bSEric Dumazet tcp_hashinfo.ehash_mask + 1, 3121fd90b29dSEric Dumazet 17, /* one slot per 128 KB of memory */ 31229e950efaSJohn Heffner 0, 31236e04e021SArnaldo Carvalho de Melo &tcp_hashinfo.bhash_size, 31241da177e4SLinus Torvalds NULL, 312531fe62b9STim Bird 0, 31261da177e4SLinus Torvalds 64 * 1024); 3127074b8517SDimitri Sivanich tcp_hashinfo.bhash_size = 1U << tcp_hashinfo.bhash_size; 31286e04e021SArnaldo Carvalho de Melo for (i = 0; i < tcp_hashinfo.bhash_size; i++) { 31296e04e021SArnaldo Carvalho de Melo spin_lock_init(&tcp_hashinfo.bhash[i].lock); 31306e04e021SArnaldo Carvalho de Melo INIT_HLIST_HEAD(&tcp_hashinfo.bhash[i].chain); 31311da177e4SLinus Torvalds } 31321da177e4SLinus Torvalds 3133c5ed63d6SEric Dumazet 3134c5ed63d6SEric Dumazet cnt = tcp_hashinfo.ehash_mask + 1; 3135c5ed63d6SEric Dumazet 3136c5ed63d6SEric Dumazet tcp_death_row.sysctl_max_tw_buckets = cnt / 2; 3137c5ed63d6SEric Dumazet sysctl_tcp_max_orphans = cnt / 2; 3138c5ed63d6SEric Dumazet sysctl_max_syn_backlog = max(128, cnt / 256); 31391da177e4SLinus Torvalds 31404acb4190SGlauber Costa tcp_init_mem(&init_net); 3141c43b874dSJason Wang /* Set per-socket limits to no more than 1/128 the pressure threshold */ 31425fb84b14SEric Dumazet limit = nr_free_buffer_pages() << (PAGE_SHIFT - 7); 3143b49960a0SEric Dumazet max_wshare = min(4UL*1024*1024, limit); 3144b49960a0SEric Dumazet max_rshare = min(6UL*1024*1024, limit); 31457b4f4b5eSJohn Heffner 31463ab224beSHideo Aoki sysctl_tcp_wmem[0] = SK_MEM_QUANTUM; 31477b4f4b5eSJohn Heffner sysctl_tcp_wmem[1] = 16*1024; 3148b49960a0SEric Dumazet sysctl_tcp_wmem[2] = max(64*1024, max_wshare); 31497b4f4b5eSJohn Heffner 31503ab224beSHideo Aoki sysctl_tcp_rmem[0] = SK_MEM_QUANTUM; 31517b4f4b5eSJohn Heffner sysctl_tcp_rmem[1] = 87380; 3152b49960a0SEric Dumazet sysctl_tcp_rmem[2] = max(87380, max_rshare); 31531da177e4SLinus Torvalds 3154afd46503SJoe Perches pr_info("Hash tables configured (established %u bind %u)\n", 3155f373b53bSEric Dumazet tcp_hashinfo.ehash_mask + 1, tcp_hashinfo.bhash_size); 3156317a76f9SStephen Hemminger 315751c5d0c4SDavid S. Miller tcp_metrics_init(); 315851c5d0c4SDavid S. Miller 3159317a76f9SStephen Hemminger tcp_register_congestion_control(&tcp_reno); 3160da5c78c8SWilliam Allen Simpson 316146d3ceabSEric Dumazet tcp_tasklet_init(); 31621da177e4SLinus Torvalds } 3163