11da177e4SLinus Torvalds /* 21da177e4SLinus Torvalds * INET An implementation of the TCP/IP protocol suite for the LINUX 31da177e4SLinus Torvalds * operating system. INET is implemented using the BSD Socket 41da177e4SLinus Torvalds * interface as the means of communication with the user level. 51da177e4SLinus Torvalds * 61da177e4SLinus Torvalds * Implementation of the Transmission Control Protocol(TCP). 71da177e4SLinus Torvalds * 802c30a84SJesper Juhl * Authors: Ross Biro 91da177e4SLinus Torvalds * Fred N. van Kempen, <waltje@uWalt.NL.Mugnet.ORG> 101da177e4SLinus Torvalds * Mark Evans, <evansmp@uhura.aston.ac.uk> 111da177e4SLinus Torvalds * Corey Minyard <wf-rch!minyard@relay.EU.net> 121da177e4SLinus Torvalds * Florian La Roche, <flla@stud.uni-sb.de> 131da177e4SLinus Torvalds * Charles Hedrick, <hedrick@klinzhai.rutgers.edu> 141da177e4SLinus Torvalds * Linus Torvalds, <torvalds@cs.helsinki.fi> 151da177e4SLinus Torvalds * Alan Cox, <gw4pts@gw4pts.ampr.org> 161da177e4SLinus Torvalds * Matthew Dillon, <dillon@apollo.west.oic.com> 171da177e4SLinus Torvalds * Arnt Gulbrandsen, <agulbra@nvg.unit.no> 181da177e4SLinus Torvalds * Jorge Cwik, <jorge@laser.satlink.net> 191da177e4SLinus Torvalds * 201da177e4SLinus Torvalds * Fixes: 211da177e4SLinus Torvalds * Alan Cox : Numerous verify_area() calls 221da177e4SLinus Torvalds * Alan Cox : Set the ACK bit on a reset 231da177e4SLinus Torvalds * Alan Cox : Stopped it crashing if it closed while 241da177e4SLinus Torvalds * sk->inuse=1 and was trying to connect 251da177e4SLinus Torvalds * (tcp_err()). 261da177e4SLinus Torvalds * Alan Cox : All icmp error handling was broken 271da177e4SLinus Torvalds * pointers passed where wrong and the 281da177e4SLinus Torvalds * socket was looked up backwards. Nobody 291da177e4SLinus Torvalds * tested any icmp error code obviously. 301da177e4SLinus Torvalds * Alan Cox : tcp_err() now handled properly. It 311da177e4SLinus Torvalds * wakes people on errors. poll 321da177e4SLinus Torvalds * behaves and the icmp error race 331da177e4SLinus Torvalds * has gone by moving it into sock.c 341da177e4SLinus Torvalds * Alan Cox : tcp_send_reset() fixed to work for 351da177e4SLinus Torvalds * everything not just packets for 361da177e4SLinus Torvalds * unknown sockets. 371da177e4SLinus Torvalds * Alan Cox : tcp option processing. 381da177e4SLinus Torvalds * Alan Cox : Reset tweaked (still not 100%) [Had 391da177e4SLinus Torvalds * syn rule wrong] 401da177e4SLinus Torvalds * Herp Rosmanith : More reset fixes 411da177e4SLinus Torvalds * Alan Cox : No longer acks invalid rst frames. 421da177e4SLinus Torvalds * Acking any kind of RST is right out. 431da177e4SLinus Torvalds * Alan Cox : Sets an ignore me flag on an rst 441da177e4SLinus Torvalds * receive otherwise odd bits of prattle 451da177e4SLinus Torvalds * escape still 461da177e4SLinus Torvalds * Alan Cox : Fixed another acking RST frame bug. 471da177e4SLinus Torvalds * Should stop LAN workplace lockups. 481da177e4SLinus Torvalds * Alan Cox : Some tidyups using the new skb list 491da177e4SLinus Torvalds * facilities 501da177e4SLinus Torvalds * Alan Cox : sk->keepopen now seems to work 511da177e4SLinus Torvalds * Alan Cox : Pulls options out correctly on accepts 521da177e4SLinus Torvalds * Alan Cox : Fixed assorted sk->rqueue->next errors 531da177e4SLinus Torvalds * Alan Cox : PSH doesn't end a TCP read. Switched a 541da177e4SLinus Torvalds * bit to skb ops. 551da177e4SLinus Torvalds * Alan Cox : Tidied tcp_data to avoid a potential 561da177e4SLinus Torvalds * nasty. 571da177e4SLinus Torvalds * Alan Cox : Added some better commenting, as the 581da177e4SLinus Torvalds * tcp is hard to follow 591da177e4SLinus Torvalds * Alan Cox : Removed incorrect check for 20 * psh 601da177e4SLinus Torvalds * Michael O'Reilly : ack < copied bug fix. 611da177e4SLinus Torvalds * Johannes Stille : Misc tcp fixes (not all in yet). 621da177e4SLinus Torvalds * Alan Cox : FIN with no memory -> CRASH 631da177e4SLinus Torvalds * Alan Cox : Added socket option proto entries. 641da177e4SLinus Torvalds * Also added awareness of them to accept. 651da177e4SLinus Torvalds * Alan Cox : Added TCP options (SOL_TCP) 661da177e4SLinus Torvalds * Alan Cox : Switched wakeup calls to callbacks, 671da177e4SLinus Torvalds * so the kernel can layer network 681da177e4SLinus Torvalds * sockets. 691da177e4SLinus Torvalds * Alan Cox : Use ip_tos/ip_ttl settings. 701da177e4SLinus Torvalds * Alan Cox : Handle FIN (more) properly (we hope). 711da177e4SLinus Torvalds * Alan Cox : RST frames sent on unsynchronised 721da177e4SLinus Torvalds * state ack error. 731da177e4SLinus Torvalds * Alan Cox : Put in missing check for SYN bit. 741da177e4SLinus Torvalds * Alan Cox : Added tcp_select_window() aka NET2E 751da177e4SLinus Torvalds * window non shrink trick. 761da177e4SLinus Torvalds * Alan Cox : Added a couple of small NET2E timer 771da177e4SLinus Torvalds * fixes 781da177e4SLinus Torvalds * Charles Hedrick : TCP fixes 791da177e4SLinus Torvalds * Toomas Tamm : TCP window fixes 801da177e4SLinus Torvalds * Alan Cox : Small URG fix to rlogin ^C ack fight 811da177e4SLinus Torvalds * Charles Hedrick : Rewrote most of it to actually work 821da177e4SLinus Torvalds * Linus : Rewrote tcp_read() and URG handling 831da177e4SLinus Torvalds * completely 841da177e4SLinus Torvalds * Gerhard Koerting: Fixed some missing timer handling 851da177e4SLinus Torvalds * Matthew Dillon : Reworked TCP machine states as per RFC 861da177e4SLinus Torvalds * Gerhard Koerting: PC/TCP workarounds 871da177e4SLinus Torvalds * Adam Caldwell : Assorted timer/timing errors 881da177e4SLinus Torvalds * Matthew Dillon : Fixed another RST bug 891da177e4SLinus Torvalds * Alan Cox : Move to kernel side addressing changes. 901da177e4SLinus Torvalds * Alan Cox : Beginning work on TCP fastpathing 911da177e4SLinus Torvalds * (not yet usable) 921da177e4SLinus Torvalds * Arnt Gulbrandsen: Turbocharged tcp_check() routine. 931da177e4SLinus Torvalds * Alan Cox : TCP fast path debugging 941da177e4SLinus Torvalds * Alan Cox : Window clamping 951da177e4SLinus Torvalds * Michael Riepe : Bug in tcp_check() 961da177e4SLinus Torvalds * Matt Dillon : More TCP improvements and RST bug fixes 971da177e4SLinus Torvalds * Matt Dillon : Yet more small nasties remove from the 981da177e4SLinus Torvalds * TCP code (Be very nice to this man if 991da177e4SLinus Torvalds * tcp finally works 100%) 8) 1001da177e4SLinus Torvalds * Alan Cox : BSD accept semantics. 1011da177e4SLinus Torvalds * Alan Cox : Reset on closedown bug. 1021da177e4SLinus Torvalds * Peter De Schrijver : ENOTCONN check missing in tcp_sendto(). 1031da177e4SLinus Torvalds * Michael Pall : Handle poll() after URG properly in 1041da177e4SLinus Torvalds * all cases. 1051da177e4SLinus Torvalds * Michael Pall : Undo the last fix in tcp_read_urg() 1061da177e4SLinus Torvalds * (multi URG PUSH broke rlogin). 1071da177e4SLinus Torvalds * Michael Pall : Fix the multi URG PUSH problem in 1081da177e4SLinus Torvalds * tcp_readable(), poll() after URG 1091da177e4SLinus Torvalds * works now. 1101da177e4SLinus Torvalds * Michael Pall : recv(...,MSG_OOB) never blocks in the 1111da177e4SLinus Torvalds * BSD api. 1121da177e4SLinus Torvalds * Alan Cox : Changed the semantics of sk->socket to 1131da177e4SLinus Torvalds * fix a race and a signal problem with 1141da177e4SLinus Torvalds * accept() and async I/O. 1151da177e4SLinus Torvalds * Alan Cox : Relaxed the rules on tcp_sendto(). 1161da177e4SLinus Torvalds * Yury Shevchuk : Really fixed accept() blocking problem. 1171da177e4SLinus Torvalds * Craig I. Hagan : Allow for BSD compatible TIME_WAIT for 1181da177e4SLinus Torvalds * clients/servers which listen in on 1191da177e4SLinus Torvalds * fixed ports. 1201da177e4SLinus Torvalds * Alan Cox : Cleaned the above up and shrank it to 1211da177e4SLinus Torvalds * a sensible code size. 1221da177e4SLinus Torvalds * Alan Cox : Self connect lockup fix. 1231da177e4SLinus Torvalds * Alan Cox : No connect to multicast. 1241da177e4SLinus Torvalds * Ross Biro : Close unaccepted children on master 1251da177e4SLinus Torvalds * socket close. 1261da177e4SLinus Torvalds * Alan Cox : Reset tracing code. 1271da177e4SLinus Torvalds * Alan Cox : Spurious resets on shutdown. 1281da177e4SLinus Torvalds * Alan Cox : Giant 15 minute/60 second timer error 1291da177e4SLinus Torvalds * Alan Cox : Small whoops in polling before an 1301da177e4SLinus Torvalds * accept. 1311da177e4SLinus Torvalds * Alan Cox : Kept the state trace facility since 1321da177e4SLinus Torvalds * it's handy for debugging. 1331da177e4SLinus Torvalds * Alan Cox : More reset handler fixes. 1341da177e4SLinus Torvalds * Alan Cox : Started rewriting the code based on 1351da177e4SLinus Torvalds * the RFC's for other useful protocol 1361da177e4SLinus Torvalds * references see: Comer, KA9Q NOS, and 1371da177e4SLinus Torvalds * for a reference on the difference 1381da177e4SLinus Torvalds * between specifications and how BSD 1391da177e4SLinus Torvalds * works see the 4.4lite source. 1401da177e4SLinus Torvalds * A.N.Kuznetsov : Don't time wait on completion of tidy 1411da177e4SLinus Torvalds * close. 1421da177e4SLinus Torvalds * Linus Torvalds : Fin/Shutdown & copied_seq changes. 1431da177e4SLinus Torvalds * Linus Torvalds : Fixed BSD port reuse to work first syn 1441da177e4SLinus Torvalds * Alan Cox : Reimplemented timers as per the RFC 1451da177e4SLinus Torvalds * and using multiple timers for sanity. 1461da177e4SLinus Torvalds * Alan Cox : Small bug fixes, and a lot of new 1471da177e4SLinus Torvalds * comments. 1481da177e4SLinus Torvalds * Alan Cox : Fixed dual reader crash by locking 1491da177e4SLinus Torvalds * the buffers (much like datagram.c) 1501da177e4SLinus Torvalds * Alan Cox : Fixed stuck sockets in probe. A probe 1511da177e4SLinus Torvalds * now gets fed up of retrying without 1521da177e4SLinus Torvalds * (even a no space) answer. 1531da177e4SLinus Torvalds * Alan Cox : Extracted closing code better 1541da177e4SLinus Torvalds * Alan Cox : Fixed the closing state machine to 1551da177e4SLinus Torvalds * resemble the RFC. 1561da177e4SLinus Torvalds * Alan Cox : More 'per spec' fixes. 1571da177e4SLinus Torvalds * Jorge Cwik : Even faster checksumming. 1581da177e4SLinus Torvalds * Alan Cox : tcp_data() doesn't ack illegal PSH 1591da177e4SLinus Torvalds * only frames. At least one pc tcp stack 1601da177e4SLinus Torvalds * generates them. 1611da177e4SLinus Torvalds * Alan Cox : Cache last socket. 1621da177e4SLinus Torvalds * Alan Cox : Per route irtt. 1631da177e4SLinus Torvalds * Matt Day : poll()->select() match BSD precisely on error 1641da177e4SLinus Torvalds * Alan Cox : New buffers 1651da177e4SLinus Torvalds * Marc Tamsky : Various sk->prot->retransmits and 1661da177e4SLinus Torvalds * sk->retransmits misupdating fixed. 1671da177e4SLinus Torvalds * Fixed tcp_write_timeout: stuck close, 1681da177e4SLinus Torvalds * and TCP syn retries gets used now. 1691da177e4SLinus Torvalds * Mark Yarvis : In tcp_read_wakeup(), don't send an 1701da177e4SLinus Torvalds * ack if state is TCP_CLOSED. 1711da177e4SLinus Torvalds * Alan Cox : Look up device on a retransmit - routes may 1721da177e4SLinus Torvalds * change. Doesn't yet cope with MSS shrink right 1731da177e4SLinus Torvalds * but it's a start! 1741da177e4SLinus Torvalds * Marc Tamsky : Closing in closing fixes. 1751da177e4SLinus Torvalds * Mike Shaver : RFC1122 verifications. 1761da177e4SLinus Torvalds * Alan Cox : rcv_saddr errors. 1771da177e4SLinus Torvalds * Alan Cox : Block double connect(). 1781da177e4SLinus Torvalds * Alan Cox : Small hooks for enSKIP. 1791da177e4SLinus Torvalds * Alexey Kuznetsov: Path MTU discovery. 1801da177e4SLinus Torvalds * Alan Cox : Support soft errors. 1811da177e4SLinus Torvalds * Alan Cox : Fix MTU discovery pathological case 1821da177e4SLinus Torvalds * when the remote claims no mtu! 1831da177e4SLinus Torvalds * Marc Tamsky : TCP_CLOSE fix. 1841da177e4SLinus Torvalds * Colin (G3TNE) : Send a reset on syn ack replies in 1851da177e4SLinus Torvalds * window but wrong (fixes NT lpd problems) 1861da177e4SLinus Torvalds * Pedro Roque : Better TCP window handling, delayed ack. 1871da177e4SLinus Torvalds * Joerg Reuter : No modification of locked buffers in 1881da177e4SLinus Torvalds * tcp_do_retransmit() 1891da177e4SLinus Torvalds * Eric Schenk : Changed receiver side silly window 1901da177e4SLinus Torvalds * avoidance algorithm to BSD style 1911da177e4SLinus Torvalds * algorithm. This doubles throughput 1921da177e4SLinus Torvalds * against machines running Solaris, 1931da177e4SLinus Torvalds * and seems to result in general 1941da177e4SLinus Torvalds * improvement. 1951da177e4SLinus Torvalds * Stefan Magdalinski : adjusted tcp_readable() to fix FIONREAD 1961da177e4SLinus Torvalds * Willy Konynenberg : Transparent proxying support. 1971da177e4SLinus Torvalds * Mike McLagan : Routing by source 1981da177e4SLinus Torvalds * Keith Owens : Do proper merging with partial SKB's in 1991da177e4SLinus Torvalds * tcp_do_sendmsg to avoid burstiness. 2001da177e4SLinus Torvalds * Eric Schenk : Fix fast close down bug with 2011da177e4SLinus Torvalds * shutdown() followed by close(). 2021da177e4SLinus Torvalds * Andi Kleen : Make poll agree with SIGIO 2031da177e4SLinus Torvalds * Salvatore Sanfilippo : Support SO_LINGER with linger == 1 and 2041da177e4SLinus Torvalds * lingertime == 0 (RFC 793 ABORT Call) 2051da177e4SLinus Torvalds * Hirokazu Takahashi : Use copy_from_user() instead of 2061da177e4SLinus Torvalds * csum_and_copy_from_user() if possible. 2071da177e4SLinus Torvalds * 2081da177e4SLinus Torvalds * This program is free software; you can redistribute it and/or 2091da177e4SLinus Torvalds * modify it under the terms of the GNU General Public License 2101da177e4SLinus Torvalds * as published by the Free Software Foundation; either version 2111da177e4SLinus Torvalds * 2 of the License, or(at your option) any later version. 2121da177e4SLinus Torvalds * 2131da177e4SLinus Torvalds * Description of States: 2141da177e4SLinus Torvalds * 2151da177e4SLinus Torvalds * TCP_SYN_SENT sent a connection request, waiting for ack 2161da177e4SLinus Torvalds * 2171da177e4SLinus Torvalds * TCP_SYN_RECV received a connection request, sent ack, 2181da177e4SLinus Torvalds * waiting for final ack in three-way handshake. 2191da177e4SLinus Torvalds * 2201da177e4SLinus Torvalds * TCP_ESTABLISHED connection established 2211da177e4SLinus Torvalds * 2221da177e4SLinus Torvalds * TCP_FIN_WAIT1 our side has shutdown, waiting to complete 2231da177e4SLinus Torvalds * transmission of remaining buffered data 2241da177e4SLinus Torvalds * 2251da177e4SLinus Torvalds * TCP_FIN_WAIT2 all buffered data sent, waiting for remote 2261da177e4SLinus Torvalds * to shutdown 2271da177e4SLinus Torvalds * 2281da177e4SLinus Torvalds * TCP_CLOSING both sides have shutdown but we still have 2291da177e4SLinus Torvalds * data we have to finish sending 2301da177e4SLinus Torvalds * 2311da177e4SLinus Torvalds * TCP_TIME_WAIT timeout to catch resent junk before entering 2321da177e4SLinus Torvalds * closed, can only be entered from FIN_WAIT2 2331da177e4SLinus Torvalds * or CLOSING. Required because the other end 2341da177e4SLinus Torvalds * may not have gotten our last ACK causing it 2351da177e4SLinus Torvalds * to retransmit the data packet (which we ignore) 2361da177e4SLinus Torvalds * 2371da177e4SLinus Torvalds * TCP_CLOSE_WAIT remote side has shutdown and is waiting for 2381da177e4SLinus Torvalds * us to finish writing our data and to shutdown 2391da177e4SLinus Torvalds * (we have to close() to move on to LAST_ACK) 2401da177e4SLinus Torvalds * 2411da177e4SLinus Torvalds * TCP_LAST_ACK out side has shutdown after remote has 2421da177e4SLinus Torvalds * shutdown. There may still be data in our 2431da177e4SLinus Torvalds * buffer that we have to finish sending 2441da177e4SLinus Torvalds * 2451da177e4SLinus Torvalds * TCP_CLOSE socket is finished 2461da177e4SLinus Torvalds */ 2471da177e4SLinus Torvalds 248afd46503SJoe Perches #define pr_fmt(fmt) "TCP: " fmt 249afd46503SJoe Perches 250172589ccSIlpo Järvinen #include <linux/kernel.h> 2511da177e4SLinus Torvalds #include <linux/module.h> 2521da177e4SLinus Torvalds #include <linux/types.h> 2531da177e4SLinus Torvalds #include <linux/fcntl.h> 2541da177e4SLinus Torvalds #include <linux/poll.h> 2551da177e4SLinus Torvalds #include <linux/init.h> 2561da177e4SLinus Torvalds #include <linux/fs.h> 2579c55e01cSJens Axboe #include <linux/skbuff.h> 25881b23b4aSAndrew Morton #include <linux/scatterlist.h> 2599c55e01cSJens Axboe #include <linux/splice.h> 2609c55e01cSJens Axboe #include <linux/net.h> 2619c55e01cSJens Axboe #include <linux/socket.h> 2621da177e4SLinus Torvalds #include <linux/random.h> 2631da177e4SLinus Torvalds #include <linux/bootmem.h> 26457413ebcSMiquel van Smoorenburg #include <linux/highmem.h> 26557413ebcSMiquel van Smoorenburg #include <linux/swap.h> 266b8059eadSDavid S. Miller #include <linux/cache.h> 267f4c50d99SHerbert Xu #include <linux/err.h> 268cfb6eeb4SYOSHIFUJI Hideaki #include <linux/crypto.h> 269da5c78c8SWilliam Allen Simpson #include <linux/time.h> 2705a0e3ad6STejun Heo #include <linux/slab.h> 2711da177e4SLinus Torvalds 2721da177e4SLinus Torvalds #include <net/icmp.h> 2731da177e4SLinus Torvalds #include <net/tcp.h> 2741da177e4SLinus Torvalds #include <net/xfrm.h> 2751da177e4SLinus Torvalds #include <net/ip.h> 2761a2449a8SChris Leech #include <net/netdma.h> 2779c55e01cSJens Axboe #include <net/sock.h> 2781da177e4SLinus Torvalds 2791da177e4SLinus Torvalds #include <asm/uaccess.h> 2801da177e4SLinus Torvalds #include <asm/ioctls.h> 2811da177e4SLinus Torvalds 282ab32ea5dSBrian Haley int sysctl_tcp_fin_timeout __read_mostly = TCP_FIN_TIMEOUT; 2831da177e4SLinus Torvalds 284dd24c001SEric Dumazet struct percpu_counter tcp_orphan_count; 2850a5578cfSArnaldo Carvalho de Melo EXPORT_SYMBOL_GPL(tcp_orphan_count); 2860a5578cfSArnaldo Carvalho de Melo 287b8059eadSDavid S. Miller int sysctl_tcp_wmem[3] __read_mostly; 288b8059eadSDavid S. Miller int sysctl_tcp_rmem[3] __read_mostly; 2891da177e4SLinus Torvalds 2901da177e4SLinus Torvalds EXPORT_SYMBOL(sysctl_tcp_rmem); 2911da177e4SLinus Torvalds EXPORT_SYMBOL(sysctl_tcp_wmem); 2921da177e4SLinus Torvalds 2938d987e5cSEric Dumazet atomic_long_t tcp_memory_allocated; /* Current allocated memory. */ 2941da177e4SLinus Torvalds EXPORT_SYMBOL(tcp_memory_allocated); 2951748376bSEric Dumazet 2961748376bSEric Dumazet /* 2971748376bSEric Dumazet * Current number of TCP sockets. 2981748376bSEric Dumazet */ 2991748376bSEric Dumazet struct percpu_counter tcp_sockets_allocated; 3001da177e4SLinus Torvalds EXPORT_SYMBOL(tcp_sockets_allocated); 3011da177e4SLinus Torvalds 3021da177e4SLinus Torvalds /* 3039c55e01cSJens Axboe * TCP splice context 3049c55e01cSJens Axboe */ 3059c55e01cSJens Axboe struct tcp_splice_state { 3069c55e01cSJens Axboe struct pipe_inode_info *pipe; 3079c55e01cSJens Axboe size_t len; 3089c55e01cSJens Axboe unsigned int flags; 3099c55e01cSJens Axboe }; 3109c55e01cSJens Axboe 3119c55e01cSJens Axboe /* 3121da177e4SLinus Torvalds * Pressure flag: try to collapse. 3131da177e4SLinus Torvalds * Technical note: it is used by multiple contexts non atomically. 3143ab224beSHideo Aoki * All the __sk_mem_schedule() is of this nature: accounting 3151da177e4SLinus Torvalds * is strict, actions are advisory and have some latency. 3161da177e4SLinus Torvalds */ 3174103f8cdSEric Dumazet int tcp_memory_pressure __read_mostly; 3181da177e4SLinus Torvalds EXPORT_SYMBOL(tcp_memory_pressure); 3191da177e4SLinus Torvalds 3205c52ba17SPavel Emelyanov void tcp_enter_memory_pressure(struct sock *sk) 3211da177e4SLinus Torvalds { 3221da177e4SLinus Torvalds if (!tcp_memory_pressure) { 3234e673444SPavel Emelyanov NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPMEMORYPRESSURES); 3241da177e4SLinus Torvalds tcp_memory_pressure = 1; 3251da177e4SLinus Torvalds } 3261da177e4SLinus Torvalds } 3271da177e4SLinus Torvalds EXPORT_SYMBOL(tcp_enter_memory_pressure); 3281da177e4SLinus Torvalds 329b103cf34SJulian Anastasov /* Convert seconds to retransmits based on initial and max timeout */ 330b103cf34SJulian Anastasov static u8 secs_to_retrans(int seconds, int timeout, int rto_max) 331b103cf34SJulian Anastasov { 332b103cf34SJulian Anastasov u8 res = 0; 333b103cf34SJulian Anastasov 334b103cf34SJulian Anastasov if (seconds > 0) { 335b103cf34SJulian Anastasov int period = timeout; 336b103cf34SJulian Anastasov 337b103cf34SJulian Anastasov res = 1; 338b103cf34SJulian Anastasov while (seconds > period && res < 255) { 339b103cf34SJulian Anastasov res++; 340b103cf34SJulian Anastasov timeout <<= 1; 341b103cf34SJulian Anastasov if (timeout > rto_max) 342b103cf34SJulian Anastasov timeout = rto_max; 343b103cf34SJulian Anastasov period += timeout; 344b103cf34SJulian Anastasov } 345b103cf34SJulian Anastasov } 346b103cf34SJulian Anastasov return res; 347b103cf34SJulian Anastasov } 348b103cf34SJulian Anastasov 349b103cf34SJulian Anastasov /* Convert retransmits to seconds based on initial and max timeout */ 350b103cf34SJulian Anastasov static int retrans_to_secs(u8 retrans, int timeout, int rto_max) 351b103cf34SJulian Anastasov { 352b103cf34SJulian Anastasov int period = 0; 353b103cf34SJulian Anastasov 354b103cf34SJulian Anastasov if (retrans > 0) { 355b103cf34SJulian Anastasov period = timeout; 356b103cf34SJulian Anastasov while (--retrans) { 357b103cf34SJulian Anastasov timeout <<= 1; 358b103cf34SJulian Anastasov if (timeout > rto_max) 359b103cf34SJulian Anastasov timeout = rto_max; 360b103cf34SJulian Anastasov period += timeout; 361b103cf34SJulian Anastasov } 362b103cf34SJulian Anastasov } 363b103cf34SJulian Anastasov return period; 364b103cf34SJulian Anastasov } 365b103cf34SJulian Anastasov 3661da177e4SLinus Torvalds /* 3671da177e4SLinus Torvalds * Wait for a TCP event. 3681da177e4SLinus Torvalds * 3691da177e4SLinus Torvalds * Note that we don't need to lock the socket, as the upper poll layers 3701da177e4SLinus Torvalds * take care of normal races (between the test and the event) and we don't 3711da177e4SLinus Torvalds * go look at any of the socket buffers directly. 3721da177e4SLinus Torvalds */ 3731da177e4SLinus Torvalds unsigned int tcp_poll(struct file *file, struct socket *sock, poll_table *wait) 3741da177e4SLinus Torvalds { 3751da177e4SLinus Torvalds unsigned int mask; 3761da177e4SLinus Torvalds struct sock *sk = sock->sk; 377cf533ea5SEric Dumazet const struct tcp_sock *tp = tcp_sk(sk); 3781da177e4SLinus Torvalds 379aa395145SEric Dumazet sock_poll_wait(file, sk_sleep(sk), wait); 3801da177e4SLinus Torvalds if (sk->sk_state == TCP_LISTEN) 381dc40c7bcSArnaldo Carvalho de Melo return inet_csk_listen_poll(sk); 3821da177e4SLinus Torvalds 3831da177e4SLinus Torvalds /* Socket is not locked. We are protected from async events 38470efce27SWill Newton * by poll logic and correct handling of state changes 38570efce27SWill Newton * made by other threads is impossible in any case. 3861da177e4SLinus Torvalds */ 3871da177e4SLinus Torvalds 3881da177e4SLinus Torvalds mask = 0; 3891da177e4SLinus Torvalds 3901da177e4SLinus Torvalds /* 3911da177e4SLinus Torvalds * POLLHUP is certainly not done right. But poll() doesn't 3921da177e4SLinus Torvalds * have a notion of HUP in just one direction, and for a 3931da177e4SLinus Torvalds * socket the read side is more interesting. 3941da177e4SLinus Torvalds * 3951da177e4SLinus Torvalds * Some poll() documentation says that POLLHUP is incompatible 3961da177e4SLinus Torvalds * with the POLLOUT/POLLWR flags, so somebody should check this 3971da177e4SLinus Torvalds * all. But careful, it tends to be safer to return too many 3981da177e4SLinus Torvalds * bits than too few, and you can easily break real applications 3991da177e4SLinus Torvalds * if you don't tell them that something has hung up! 4001da177e4SLinus Torvalds * 4011da177e4SLinus Torvalds * Check-me. 4021da177e4SLinus Torvalds * 4031da177e4SLinus Torvalds * Check number 1. POLLHUP is _UNMASKABLE_ event (see UNIX98 and 4041da177e4SLinus Torvalds * our fs/select.c). It means that after we received EOF, 4051da177e4SLinus Torvalds * poll always returns immediately, making impossible poll() on write() 4061da177e4SLinus Torvalds * in state CLOSE_WAIT. One solution is evident --- to set POLLHUP 4071da177e4SLinus Torvalds * if and only if shutdown has been made in both directions. 4081da177e4SLinus Torvalds * Actually, it is interesting to look how Solaris and DUX 40970efce27SWill Newton * solve this dilemma. I would prefer, if POLLHUP were maskable, 4101da177e4SLinus Torvalds * then we could set it on SND_SHUTDOWN. BTW examples given 4111da177e4SLinus Torvalds * in Stevens' books assume exactly this behaviour, it explains 41270efce27SWill Newton * why POLLHUP is incompatible with POLLOUT. --ANK 4131da177e4SLinus Torvalds * 4141da177e4SLinus Torvalds * NOTE. Check for TCP_CLOSE is added. The goal is to prevent 4151da177e4SLinus Torvalds * blocking on fresh not-connected or disconnected socket. --ANK 4161da177e4SLinus Torvalds */ 4171da177e4SLinus Torvalds if (sk->sk_shutdown == SHUTDOWN_MASK || sk->sk_state == TCP_CLOSE) 4181da177e4SLinus Torvalds mask |= POLLHUP; 4191da177e4SLinus Torvalds if (sk->sk_shutdown & RCV_SHUTDOWN) 420f348d70aSDavide Libenzi mask |= POLLIN | POLLRDNORM | POLLRDHUP; 4211da177e4SLinus Torvalds 4221da177e4SLinus Torvalds /* Connected? */ 4231da177e4SLinus Torvalds if ((1 << sk->sk_state) & ~(TCPF_SYN_SENT | TCPF_SYN_RECV)) { 424c7004482SDavid S. Miller int target = sock_rcvlowat(sk, 0, INT_MAX); 425c7004482SDavid S. Miller 426c7004482SDavid S. Miller if (tp->urg_seq == tp->copied_seq && 427c7004482SDavid S. Miller !sock_flag(sk, SOCK_URGINLINE) && 428c7004482SDavid S. Miller tp->urg_data) 429b634f875SAlexandra Kossovsky target++; 430c7004482SDavid S. Miller 4311da177e4SLinus Torvalds /* Potential race condition. If read of tp below will 4321da177e4SLinus Torvalds * escape above sk->sk_state, we can be illegally awaken 4331da177e4SLinus Torvalds * in SYN_* states. */ 434c7004482SDavid S. Miller if (tp->rcv_nxt - tp->copied_seq >= target) 4351da177e4SLinus Torvalds mask |= POLLIN | POLLRDNORM; 4361da177e4SLinus Torvalds 4371da177e4SLinus Torvalds if (!(sk->sk_shutdown & SEND_SHUTDOWN)) { 4381da177e4SLinus Torvalds if (sk_stream_wspace(sk) >= sk_stream_min_wspace(sk)) { 4391da177e4SLinus Torvalds mask |= POLLOUT | POLLWRNORM; 4401da177e4SLinus Torvalds } else { /* send SIGIO later */ 4411da177e4SLinus Torvalds set_bit(SOCK_ASYNC_NOSPACE, 4421da177e4SLinus Torvalds &sk->sk_socket->flags); 4431da177e4SLinus Torvalds set_bit(SOCK_NOSPACE, &sk->sk_socket->flags); 4441da177e4SLinus Torvalds 4451da177e4SLinus Torvalds /* Race breaker. If space is freed after 4461da177e4SLinus Torvalds * wspace test but before the flags are set, 4471da177e4SLinus Torvalds * IO signal will be lost. 4481da177e4SLinus Torvalds */ 4491da177e4SLinus Torvalds if (sk_stream_wspace(sk) >= sk_stream_min_wspace(sk)) 4501da177e4SLinus Torvalds mask |= POLLOUT | POLLWRNORM; 4511da177e4SLinus Torvalds } 452d84ba638SKOSAKI Motohiro } else 453d84ba638SKOSAKI Motohiro mask |= POLLOUT | POLLWRNORM; 4541da177e4SLinus Torvalds 4551da177e4SLinus Torvalds if (tp->urg_data & TCP_URG_VALID) 4561da177e4SLinus Torvalds mask |= POLLPRI; 4571da177e4SLinus Torvalds } 458a4d25803STom Marshall /* This barrier is coupled with smp_wmb() in tcp_reset() */ 459a4d25803STom Marshall smp_rmb(); 460a4d25803STom Marshall if (sk->sk_err) 461a4d25803STom Marshall mask |= POLLERR; 462a4d25803STom Marshall 4631da177e4SLinus Torvalds return mask; 4641da177e4SLinus Torvalds } 4654bc2f18bSEric Dumazet EXPORT_SYMBOL(tcp_poll); 4661da177e4SLinus Torvalds 4671da177e4SLinus Torvalds int tcp_ioctl(struct sock *sk, int cmd, unsigned long arg) 4681da177e4SLinus Torvalds { 4691da177e4SLinus Torvalds struct tcp_sock *tp = tcp_sk(sk); 4701da177e4SLinus Torvalds int answ; 4711da177e4SLinus Torvalds 4721da177e4SLinus Torvalds switch (cmd) { 4731da177e4SLinus Torvalds case SIOCINQ: 4741da177e4SLinus Torvalds if (sk->sk_state == TCP_LISTEN) 4751da177e4SLinus Torvalds return -EINVAL; 4761da177e4SLinus Torvalds 4771da177e4SLinus Torvalds lock_sock(sk); 4781da177e4SLinus Torvalds if ((1 << sk->sk_state) & (TCPF_SYN_SENT | TCPF_SYN_RECV)) 4791da177e4SLinus Torvalds answ = 0; 4801da177e4SLinus Torvalds else if (sock_flag(sk, SOCK_URGINLINE) || 4811da177e4SLinus Torvalds !tp->urg_data || 4821da177e4SLinus Torvalds before(tp->urg_seq, tp->copied_seq) || 4831da177e4SLinus Torvalds !before(tp->urg_seq, tp->rcv_nxt)) { 48491521944SDavid S. Miller struct sk_buff *skb; 48591521944SDavid S. Miller 4861da177e4SLinus Torvalds answ = tp->rcv_nxt - tp->copied_seq; 4871da177e4SLinus Torvalds 4881da177e4SLinus Torvalds /* Subtract 1, if FIN is in queue. */ 48991521944SDavid S. Miller skb = skb_peek_tail(&sk->sk_receive_queue); 49091521944SDavid S. Miller if (answ && skb) 49191521944SDavid S. Miller answ -= tcp_hdr(skb)->fin; 4921da177e4SLinus Torvalds } else 4931da177e4SLinus Torvalds answ = tp->urg_seq - tp->copied_seq; 4941da177e4SLinus Torvalds release_sock(sk); 4951da177e4SLinus Torvalds break; 4961da177e4SLinus Torvalds case SIOCATMARK: 4971da177e4SLinus Torvalds answ = tp->urg_data && tp->urg_seq == tp->copied_seq; 4981da177e4SLinus Torvalds break; 4991da177e4SLinus Torvalds case SIOCOUTQ: 5001da177e4SLinus Torvalds if (sk->sk_state == TCP_LISTEN) 5011da177e4SLinus Torvalds return -EINVAL; 5021da177e4SLinus Torvalds 5031da177e4SLinus Torvalds if ((1 << sk->sk_state) & (TCPF_SYN_SENT | TCPF_SYN_RECV)) 5041da177e4SLinus Torvalds answ = 0; 5051da177e4SLinus Torvalds else 5061da177e4SLinus Torvalds answ = tp->write_seq - tp->snd_una; 5071da177e4SLinus Torvalds break; 5082f4e1b39SMario Schuknecht case SIOCOUTQNSD: 5092f4e1b39SMario Schuknecht if (sk->sk_state == TCP_LISTEN) 5102f4e1b39SMario Schuknecht return -EINVAL; 5112f4e1b39SMario Schuknecht 5122f4e1b39SMario Schuknecht if ((1 << sk->sk_state) & (TCPF_SYN_SENT | TCPF_SYN_RECV)) 5132f4e1b39SMario Schuknecht answ = 0; 5142f4e1b39SMario Schuknecht else 5152f4e1b39SMario Schuknecht answ = tp->write_seq - tp->snd_nxt; 5162f4e1b39SMario Schuknecht break; 5171da177e4SLinus Torvalds default: 5181da177e4SLinus Torvalds return -ENOIOCTLCMD; 5193ff50b79SStephen Hemminger } 5201da177e4SLinus Torvalds 5211da177e4SLinus Torvalds return put_user(answ, (int __user *)arg); 5221da177e4SLinus Torvalds } 5234bc2f18bSEric Dumazet EXPORT_SYMBOL(tcp_ioctl); 5241da177e4SLinus Torvalds 5251da177e4SLinus Torvalds static inline void tcp_mark_push(struct tcp_sock *tp, struct sk_buff *skb) 5261da177e4SLinus Torvalds { 5274de075e0SEric Dumazet TCP_SKB_CB(skb)->tcp_flags |= TCPHDR_PSH; 5281da177e4SLinus Torvalds tp->pushed_seq = tp->write_seq; 5291da177e4SLinus Torvalds } 5301da177e4SLinus Torvalds 531cf533ea5SEric Dumazet static inline int forced_push(const struct tcp_sock *tp) 5321da177e4SLinus Torvalds { 5331da177e4SLinus Torvalds return after(tp->write_seq, tp->pushed_seq + (tp->max_window >> 1)); 5341da177e4SLinus Torvalds } 5351da177e4SLinus Torvalds 5369e412ba7SIlpo Järvinen static inline void skb_entail(struct sock *sk, struct sk_buff *skb) 5371da177e4SLinus Torvalds { 5389e412ba7SIlpo Järvinen struct tcp_sock *tp = tcp_sk(sk); 539352d4800SArnaldo Carvalho de Melo struct tcp_skb_cb *tcb = TCP_SKB_CB(skb); 540352d4800SArnaldo Carvalho de Melo 5411da177e4SLinus Torvalds skb->csum = 0; 542352d4800SArnaldo Carvalho de Melo tcb->seq = tcb->end_seq = tp->write_seq; 5434de075e0SEric Dumazet tcb->tcp_flags = TCPHDR_ACK; 544352d4800SArnaldo Carvalho de Melo tcb->sacked = 0; 5451da177e4SLinus Torvalds skb_header_release(skb); 546fe067e8aSDavid S. Miller tcp_add_write_queue_tail(sk, skb); 5473ab224beSHideo Aoki sk->sk_wmem_queued += skb->truesize; 5483ab224beSHideo Aoki sk_mem_charge(sk, skb->truesize); 54989ebd197SDavid S. Miller if (tp->nonagle & TCP_NAGLE_PUSH) 5501da177e4SLinus Torvalds tp->nonagle &= ~TCP_NAGLE_PUSH; 5511da177e4SLinus Torvalds } 5521da177e4SLinus Torvalds 553afeca340SKrishna Kumar static inline void tcp_mark_urg(struct tcp_sock *tp, int flags) 5541da177e4SLinus Torvalds { 55533f5f57eSIlpo Järvinen if (flags & MSG_OOB) 5561da177e4SLinus Torvalds tp->snd_up = tp->write_seq; 5571da177e4SLinus Torvalds } 5581da177e4SLinus Torvalds 5599e412ba7SIlpo Järvinen static inline void tcp_push(struct sock *sk, int flags, int mss_now, 5609e412ba7SIlpo Järvinen int nonagle) 5611da177e4SLinus Torvalds { 562afeca340SKrishna Kumar if (tcp_send_head(sk)) { 5639e412ba7SIlpo Järvinen struct tcp_sock *tp = tcp_sk(sk); 5649e412ba7SIlpo Järvinen 5651da177e4SLinus Torvalds if (!(flags & MSG_MORE) || forced_push(tp)) 566afeca340SKrishna Kumar tcp_mark_push(tp, tcp_write_queue_tail(sk)); 567afeca340SKrishna Kumar 568afeca340SKrishna Kumar tcp_mark_urg(tp, flags); 5699e412ba7SIlpo Järvinen __tcp_push_pending_frames(sk, mss_now, 5701da177e4SLinus Torvalds (flags & MSG_MORE) ? TCP_NAGLE_CORK : nonagle); 5711da177e4SLinus Torvalds } 5721da177e4SLinus Torvalds } 5731da177e4SLinus Torvalds 5746ff7751dSAdrian Bunk static int tcp_splice_data_recv(read_descriptor_t *rd_desc, struct sk_buff *skb, 5759c55e01cSJens Axboe unsigned int offset, size_t len) 5769c55e01cSJens Axboe { 5779c55e01cSJens Axboe struct tcp_splice_state *tss = rd_desc->arg.data; 57833966dd0SWilly Tarreau int ret; 5799c55e01cSJens Axboe 5809fa5fdf2SDimitris Michailidis ret = skb_splice_bits(skb, offset, tss->pipe, min(rd_desc->count, len), 5819fa5fdf2SDimitris Michailidis tss->flags); 58233966dd0SWilly Tarreau if (ret > 0) 58333966dd0SWilly Tarreau rd_desc->count -= ret; 58433966dd0SWilly Tarreau return ret; 5859c55e01cSJens Axboe } 5869c55e01cSJens Axboe 5879c55e01cSJens Axboe static int __tcp_splice_read(struct sock *sk, struct tcp_splice_state *tss) 5889c55e01cSJens Axboe { 5899c55e01cSJens Axboe /* Store TCP splice context information in read_descriptor_t. */ 5909c55e01cSJens Axboe read_descriptor_t rd_desc = { 5919c55e01cSJens Axboe .arg.data = tss, 59233966dd0SWilly Tarreau .count = tss->len, 5939c55e01cSJens Axboe }; 5949c55e01cSJens Axboe 5959c55e01cSJens Axboe return tcp_read_sock(sk, &rd_desc, tcp_splice_data_recv); 5969c55e01cSJens Axboe } 5979c55e01cSJens Axboe 5989c55e01cSJens Axboe /** 5999c55e01cSJens Axboe * tcp_splice_read - splice data from TCP socket to a pipe 6009c55e01cSJens Axboe * @sock: socket to splice from 6019c55e01cSJens Axboe * @ppos: position (not valid) 6029c55e01cSJens Axboe * @pipe: pipe to splice to 6039c55e01cSJens Axboe * @len: number of bytes to splice 6049c55e01cSJens Axboe * @flags: splice modifier flags 6059c55e01cSJens Axboe * 6069c55e01cSJens Axboe * Description: 6079c55e01cSJens Axboe * Will read pages from given socket and fill them into a pipe. 6089c55e01cSJens Axboe * 6099c55e01cSJens Axboe **/ 6109c55e01cSJens Axboe ssize_t tcp_splice_read(struct socket *sock, loff_t *ppos, 6119c55e01cSJens Axboe struct pipe_inode_info *pipe, size_t len, 6129c55e01cSJens Axboe unsigned int flags) 6139c55e01cSJens Axboe { 6149c55e01cSJens Axboe struct sock *sk = sock->sk; 6159c55e01cSJens Axboe struct tcp_splice_state tss = { 6169c55e01cSJens Axboe .pipe = pipe, 6179c55e01cSJens Axboe .len = len, 6189c55e01cSJens Axboe .flags = flags, 6199c55e01cSJens Axboe }; 6209c55e01cSJens Axboe long timeo; 6219c55e01cSJens Axboe ssize_t spliced; 6229c55e01cSJens Axboe int ret; 6239c55e01cSJens Axboe 6243a047bf8SChangli Gao sock_rps_record_flow(sk); 6259c55e01cSJens Axboe /* 6269c55e01cSJens Axboe * We can't seek on a socket input 6279c55e01cSJens Axboe */ 6289c55e01cSJens Axboe if (unlikely(*ppos)) 6299c55e01cSJens Axboe return -ESPIPE; 6309c55e01cSJens Axboe 6319c55e01cSJens Axboe ret = spliced = 0; 6329c55e01cSJens Axboe 6339c55e01cSJens Axboe lock_sock(sk); 6349c55e01cSJens Axboe 63542324c62SEric Dumazet timeo = sock_rcvtimeo(sk, sock->file->f_flags & O_NONBLOCK); 6369c55e01cSJens Axboe while (tss.len) { 6379c55e01cSJens Axboe ret = __tcp_splice_read(sk, &tss); 6389c55e01cSJens Axboe if (ret < 0) 6399c55e01cSJens Axboe break; 6409c55e01cSJens Axboe else if (!ret) { 6419c55e01cSJens Axboe if (spliced) 6429c55e01cSJens Axboe break; 6439c55e01cSJens Axboe if (sock_flag(sk, SOCK_DONE)) 6449c55e01cSJens Axboe break; 6459c55e01cSJens Axboe if (sk->sk_err) { 6469c55e01cSJens Axboe ret = sock_error(sk); 6479c55e01cSJens Axboe break; 6489c55e01cSJens Axboe } 6499c55e01cSJens Axboe if (sk->sk_shutdown & RCV_SHUTDOWN) 6509c55e01cSJens Axboe break; 6519c55e01cSJens Axboe if (sk->sk_state == TCP_CLOSE) { 6529c55e01cSJens Axboe /* 6539c55e01cSJens Axboe * This occurs when user tries to read 6549c55e01cSJens Axboe * from never connected socket. 6559c55e01cSJens Axboe */ 6569c55e01cSJens Axboe if (!sock_flag(sk, SOCK_DONE)) 6579c55e01cSJens Axboe ret = -ENOTCONN; 6589c55e01cSJens Axboe break; 6599c55e01cSJens Axboe } 6609c55e01cSJens Axboe if (!timeo) { 6619c55e01cSJens Axboe ret = -EAGAIN; 6629c55e01cSJens Axboe break; 6639c55e01cSJens Axboe } 6649c55e01cSJens Axboe sk_wait_data(sk, &timeo); 6659c55e01cSJens Axboe if (signal_pending(current)) { 6669c55e01cSJens Axboe ret = sock_intr_errno(timeo); 6679c55e01cSJens Axboe break; 6689c55e01cSJens Axboe } 6699c55e01cSJens Axboe continue; 6709c55e01cSJens Axboe } 6719c55e01cSJens Axboe tss.len -= ret; 6729c55e01cSJens Axboe spliced += ret; 6739c55e01cSJens Axboe 67433966dd0SWilly Tarreau if (!timeo) 67533966dd0SWilly Tarreau break; 6769c55e01cSJens Axboe release_sock(sk); 6779c55e01cSJens Axboe lock_sock(sk); 6789c55e01cSJens Axboe 6799c55e01cSJens Axboe if (sk->sk_err || sk->sk_state == TCP_CLOSE || 68033966dd0SWilly Tarreau (sk->sk_shutdown & RCV_SHUTDOWN) || 6819c55e01cSJens Axboe signal_pending(current)) 6829c55e01cSJens Axboe break; 6839c55e01cSJens Axboe } 6849c55e01cSJens Axboe 6859c55e01cSJens Axboe release_sock(sk); 6869c55e01cSJens Axboe 6879c55e01cSJens Axboe if (spliced) 6889c55e01cSJens Axboe return spliced; 6899c55e01cSJens Axboe 6909c55e01cSJens Axboe return ret; 6919c55e01cSJens Axboe } 6924bc2f18bSEric Dumazet EXPORT_SYMBOL(tcp_splice_read); 6939c55e01cSJens Axboe 694df97c708SPavel Emelyanov struct sk_buff *sk_stream_alloc_skb(struct sock *sk, int size, gfp_t gfp) 695f561d0f2SPavel Emelyanov { 696f561d0f2SPavel Emelyanov struct sk_buff *skb; 697f561d0f2SPavel Emelyanov 698f561d0f2SPavel Emelyanov /* The TCP header must be at least 32-bit aligned. */ 699f561d0f2SPavel Emelyanov size = ALIGN(size, 4); 700f561d0f2SPavel Emelyanov 701f561d0f2SPavel Emelyanov skb = alloc_skb_fclone(size + sk->sk_prot->max_header, gfp); 702f561d0f2SPavel Emelyanov if (skb) { 7033ab224beSHideo Aoki if (sk_wmem_schedule(sk, skb->truesize)) { 704a21d4572SEric Dumazet skb_reserve(skb, sk->sk_prot->max_header); 705f561d0f2SPavel Emelyanov /* 706f561d0f2SPavel Emelyanov * Make sure that we have exactly size bytes 707f561d0f2SPavel Emelyanov * available to the caller, no more, no less. 708f561d0f2SPavel Emelyanov */ 709a21d4572SEric Dumazet skb->avail_size = size; 710f561d0f2SPavel Emelyanov return skb; 711f561d0f2SPavel Emelyanov } 712f561d0f2SPavel Emelyanov __kfree_skb(skb); 713f561d0f2SPavel Emelyanov } else { 7145c52ba17SPavel Emelyanov sk->sk_prot->enter_memory_pressure(sk); 715f561d0f2SPavel Emelyanov sk_stream_moderate_sndbuf(sk); 716f561d0f2SPavel Emelyanov } 717f561d0f2SPavel Emelyanov return NULL; 718f561d0f2SPavel Emelyanov } 719f561d0f2SPavel Emelyanov 7200c54b85fSIlpo Järvinen static unsigned int tcp_xmit_size_goal(struct sock *sk, u32 mss_now, 7210c54b85fSIlpo Järvinen int large_allowed) 7220c54b85fSIlpo Järvinen { 7230c54b85fSIlpo Järvinen struct tcp_sock *tp = tcp_sk(sk); 7242a3a041cSIlpo Järvinen u32 xmit_size_goal, old_size_goal; 7250c54b85fSIlpo Järvinen 7260c54b85fSIlpo Järvinen xmit_size_goal = mss_now; 7270c54b85fSIlpo Järvinen 7280c54b85fSIlpo Järvinen if (large_allowed && sk_can_gso(sk)) { 7290c54b85fSIlpo Järvinen xmit_size_goal = ((sk->sk_gso_max_size - 1) - 7300c54b85fSIlpo Järvinen inet_csk(sk)->icsk_af_ops->net_header_len - 7310c54b85fSIlpo Järvinen inet_csk(sk)->icsk_ext_hdr_len - 7320c54b85fSIlpo Järvinen tp->tcp_header_len); 7330c54b85fSIlpo Järvinen 7340c54b85fSIlpo Järvinen xmit_size_goal = tcp_bound_to_half_wnd(tp, xmit_size_goal); 7352a3a041cSIlpo Järvinen 7362a3a041cSIlpo Järvinen /* We try hard to avoid divides here */ 7372a3a041cSIlpo Järvinen old_size_goal = tp->xmit_size_goal_segs * mss_now; 7382a3a041cSIlpo Järvinen 7392a3a041cSIlpo Järvinen if (likely(old_size_goal <= xmit_size_goal && 7402a3a041cSIlpo Järvinen old_size_goal + mss_now > xmit_size_goal)) { 7412a3a041cSIlpo Järvinen xmit_size_goal = old_size_goal; 7422a3a041cSIlpo Järvinen } else { 7432a3a041cSIlpo Järvinen tp->xmit_size_goal_segs = xmit_size_goal / mss_now; 7442a3a041cSIlpo Järvinen xmit_size_goal = tp->xmit_size_goal_segs * mss_now; 7452a3a041cSIlpo Järvinen } 7460c54b85fSIlpo Järvinen } 7470c54b85fSIlpo Järvinen 748afece1c6SIlpo Järvinen return max(xmit_size_goal, mss_now); 7490c54b85fSIlpo Järvinen } 7500c54b85fSIlpo Järvinen 7510c54b85fSIlpo Järvinen static int tcp_send_mss(struct sock *sk, int *size_goal, int flags) 7520c54b85fSIlpo Järvinen { 7530c54b85fSIlpo Järvinen int mss_now; 7540c54b85fSIlpo Järvinen 7550c54b85fSIlpo Järvinen mss_now = tcp_current_mss(sk); 7560c54b85fSIlpo Järvinen *size_goal = tcp_xmit_size_goal(sk, mss_now, !(flags & MSG_OOB)); 7570c54b85fSIlpo Järvinen 7580c54b85fSIlpo Järvinen return mss_now; 7590c54b85fSIlpo Järvinen } 7600c54b85fSIlpo Järvinen 7611da177e4SLinus Torvalds static ssize_t do_tcp_sendpages(struct sock *sk, struct page **pages, int poffset, 7621da177e4SLinus Torvalds size_t psize, int flags) 7631da177e4SLinus Torvalds { 7641da177e4SLinus Torvalds struct tcp_sock *tp = tcp_sk(sk); 765c1b4a7e6SDavid S. Miller int mss_now, size_goal; 7661da177e4SLinus Torvalds int err; 7671da177e4SLinus Torvalds ssize_t copied; 7681da177e4SLinus Torvalds long timeo = sock_sndtimeo(sk, flags & MSG_DONTWAIT); 7691da177e4SLinus Torvalds 7701da177e4SLinus Torvalds /* Wait for a connection to finish. */ 7711da177e4SLinus Torvalds if ((1 << sk->sk_state) & ~(TCPF_ESTABLISHED | TCPF_CLOSE_WAIT)) 7721da177e4SLinus Torvalds if ((err = sk_stream_wait_connect(sk, &timeo)) != 0) 7731da177e4SLinus Torvalds goto out_err; 7741da177e4SLinus Torvalds 7751da177e4SLinus Torvalds clear_bit(SOCK_ASYNC_NOSPACE, &sk->sk_socket->flags); 7761da177e4SLinus Torvalds 7770c54b85fSIlpo Järvinen mss_now = tcp_send_mss(sk, &size_goal, flags); 7781da177e4SLinus Torvalds copied = 0; 7791da177e4SLinus Torvalds 7801da177e4SLinus Torvalds err = -EPIPE; 7811da177e4SLinus Torvalds if (sk->sk_err || (sk->sk_shutdown & SEND_SHUTDOWN)) 7820d6a775eSIlpo Järvinen goto out_err; 7831da177e4SLinus Torvalds 7841da177e4SLinus Torvalds while (psize > 0) { 785fe067e8aSDavid S. Miller struct sk_buff *skb = tcp_write_queue_tail(sk); 7861da177e4SLinus Torvalds struct page *page = pages[poffset / PAGE_SIZE]; 7871da177e4SLinus Torvalds int copy, i, can_coalesce; 7881da177e4SLinus Torvalds int offset = poffset % PAGE_SIZE; 7891da177e4SLinus Torvalds int size = min_t(size_t, psize, PAGE_SIZE - offset); 7901da177e4SLinus Torvalds 791fe067e8aSDavid S. Miller if (!tcp_send_head(sk) || (copy = size_goal - skb->len) <= 0) { 7921da177e4SLinus Torvalds new_segment: 7931da177e4SLinus Torvalds if (!sk_stream_memory_free(sk)) 7941da177e4SLinus Torvalds goto wait_for_sndbuf; 7951da177e4SLinus Torvalds 796df97c708SPavel Emelyanov skb = sk_stream_alloc_skb(sk, 0, sk->sk_allocation); 7971da177e4SLinus Torvalds if (!skb) 7981da177e4SLinus Torvalds goto wait_for_memory; 7991da177e4SLinus Torvalds 8009e412ba7SIlpo Järvinen skb_entail(sk, skb); 801c1b4a7e6SDavid S. Miller copy = size_goal; 8021da177e4SLinus Torvalds } 8031da177e4SLinus Torvalds 8041da177e4SLinus Torvalds if (copy > size) 8051da177e4SLinus Torvalds copy = size; 8061da177e4SLinus Torvalds 8071da177e4SLinus Torvalds i = skb_shinfo(skb)->nr_frags; 8081da177e4SLinus Torvalds can_coalesce = skb_can_coalesce(skb, i, page, offset); 8091da177e4SLinus Torvalds if (!can_coalesce && i >= MAX_SKB_FRAGS) { 8101da177e4SLinus Torvalds tcp_mark_push(tp, skb); 8111da177e4SLinus Torvalds goto new_segment; 8121da177e4SLinus Torvalds } 8133ab224beSHideo Aoki if (!sk_wmem_schedule(sk, copy)) 8141da177e4SLinus Torvalds goto wait_for_memory; 8151da177e4SLinus Torvalds 8161da177e4SLinus Torvalds if (can_coalesce) { 8179e903e08SEric Dumazet skb_frag_size_add(&skb_shinfo(skb)->frags[i - 1], copy); 8181da177e4SLinus Torvalds } else { 8191da177e4SLinus Torvalds get_page(page); 8201da177e4SLinus Torvalds skb_fill_page_desc(skb, i, page, offset, copy); 8211da177e4SLinus Torvalds } 8221da177e4SLinus Torvalds 8231da177e4SLinus Torvalds skb->len += copy; 8241da177e4SLinus Torvalds skb->data_len += copy; 8251da177e4SLinus Torvalds skb->truesize += copy; 8261da177e4SLinus Torvalds sk->sk_wmem_queued += copy; 8273ab224beSHideo Aoki sk_mem_charge(sk, copy); 82884fa7933SPatrick McHardy skb->ip_summed = CHECKSUM_PARTIAL; 8291da177e4SLinus Torvalds tp->write_seq += copy; 8301da177e4SLinus Torvalds TCP_SKB_CB(skb)->end_seq += copy; 8317967168cSHerbert Xu skb_shinfo(skb)->gso_segs = 0; 8321da177e4SLinus Torvalds 8331da177e4SLinus Torvalds if (!copied) 8344de075e0SEric Dumazet TCP_SKB_CB(skb)->tcp_flags &= ~TCPHDR_PSH; 8351da177e4SLinus Torvalds 8361da177e4SLinus Torvalds copied += copy; 8371da177e4SLinus Torvalds poffset += copy; 8381da177e4SLinus Torvalds if (!(psize -= copy)) 8391da177e4SLinus Torvalds goto out; 8401da177e4SLinus Torvalds 84169d15067SHerbert Xu if (skb->len < size_goal || (flags & MSG_OOB)) 8421da177e4SLinus Torvalds continue; 8431da177e4SLinus Torvalds 8441da177e4SLinus Torvalds if (forced_push(tp)) { 8451da177e4SLinus Torvalds tcp_mark_push(tp, skb); 8469e412ba7SIlpo Järvinen __tcp_push_pending_frames(sk, mss_now, TCP_NAGLE_PUSH); 847fe067e8aSDavid S. Miller } else if (skb == tcp_send_head(sk)) 8481da177e4SLinus Torvalds tcp_push_one(sk, mss_now); 8491da177e4SLinus Torvalds continue; 8501da177e4SLinus Torvalds 8511da177e4SLinus Torvalds wait_for_sndbuf: 8521da177e4SLinus Torvalds set_bit(SOCK_NOSPACE, &sk->sk_socket->flags); 8531da177e4SLinus Torvalds wait_for_memory: 8541da177e4SLinus Torvalds if (copied) 8559e412ba7SIlpo Järvinen tcp_push(sk, flags & ~MSG_MORE, mss_now, TCP_NAGLE_PUSH); 8561da177e4SLinus Torvalds 8571da177e4SLinus Torvalds if ((err = sk_stream_wait_memory(sk, &timeo)) != 0) 8581da177e4SLinus Torvalds goto do_error; 8591da177e4SLinus Torvalds 8600c54b85fSIlpo Järvinen mss_now = tcp_send_mss(sk, &size_goal, flags); 8611da177e4SLinus Torvalds } 8621da177e4SLinus Torvalds 8631da177e4SLinus Torvalds out: 86435f9c09fSEric Dumazet if (copied && !(flags & MSG_SENDPAGE_NOTLAST)) 8659e412ba7SIlpo Järvinen tcp_push(sk, flags, mss_now, tp->nonagle); 8661da177e4SLinus Torvalds return copied; 8671da177e4SLinus Torvalds 8681da177e4SLinus Torvalds do_error: 8691da177e4SLinus Torvalds if (copied) 8701da177e4SLinus Torvalds goto out; 8711da177e4SLinus Torvalds out_err: 8721da177e4SLinus Torvalds return sk_stream_error(sk, flags, err); 8731da177e4SLinus Torvalds } 8741da177e4SLinus Torvalds 8757ba42910SChangli Gao int tcp_sendpage(struct sock *sk, struct page *page, int offset, 8761da177e4SLinus Torvalds size_t size, int flags) 8771da177e4SLinus Torvalds { 8781da177e4SLinus Torvalds ssize_t res; 8791da177e4SLinus Torvalds 8801da177e4SLinus Torvalds if (!(sk->sk_route_caps & NETIF_F_SG) || 8818648b305SHerbert Xu !(sk->sk_route_caps & NETIF_F_ALL_CSUM)) 8827ba42910SChangli Gao return sock_no_sendpage(sk->sk_socket, page, offset, size, 8837ba42910SChangli Gao flags); 8841da177e4SLinus Torvalds 8851da177e4SLinus Torvalds lock_sock(sk); 8861da177e4SLinus Torvalds res = do_tcp_sendpages(sk, &page, offset, size, flags); 8871da177e4SLinus Torvalds release_sock(sk); 8881da177e4SLinus Torvalds return res; 8891da177e4SLinus Torvalds } 8904bc2f18bSEric Dumazet EXPORT_SYMBOL(tcp_sendpage); 8911da177e4SLinus Torvalds 892690e99c4SEric Dumazet static inline int select_size(const struct sock *sk, bool sg) 8931da177e4SLinus Torvalds { 894cf533ea5SEric Dumazet const struct tcp_sock *tp = tcp_sk(sk); 895c1b4a7e6SDavid S. Miller int tmp = tp->mss_cache; 8961da177e4SLinus Torvalds 897def87cf4SKrishna Kumar if (sg) { 898f07d960dSEric Dumazet if (sk_can_gso(sk)) { 899f07d960dSEric Dumazet /* Small frames wont use a full page: 900f07d960dSEric Dumazet * Payload will immediately follow tcp header. 901f07d960dSEric Dumazet */ 902f07d960dSEric Dumazet tmp = SKB_WITH_OVERHEAD(2048 - MAX_TCP_HEADER); 903f07d960dSEric Dumazet } else { 904b4e26f5eSDavid S. Miller int pgbreak = SKB_MAX_HEAD(MAX_TCP_HEADER); 905b4e26f5eSDavid S. Miller 906b4e26f5eSDavid S. Miller if (tmp >= pgbreak && 907b4e26f5eSDavid S. Miller tmp <= pgbreak + (MAX_SKB_FRAGS - 1) * PAGE_SIZE) 908b4e26f5eSDavid S. Miller tmp = pgbreak; 909b4e26f5eSDavid S. Miller } 910b4e26f5eSDavid S. Miller } 9111da177e4SLinus Torvalds 9121da177e4SLinus Torvalds return tmp; 9131da177e4SLinus Torvalds } 9141da177e4SLinus Torvalds 9157ba42910SChangli Gao int tcp_sendmsg(struct kiocb *iocb, struct sock *sk, struct msghdr *msg, 9161da177e4SLinus Torvalds size_t size) 9171da177e4SLinus Torvalds { 9181da177e4SLinus Torvalds struct iovec *iov; 9191da177e4SLinus Torvalds struct tcp_sock *tp = tcp_sk(sk); 9201da177e4SLinus Torvalds struct sk_buff *skb; 921690e99c4SEric Dumazet int iovlen, flags, err, copied; 922370816aeSPavel Emelyanov int mss_now = 0, size_goal; 923690e99c4SEric Dumazet bool sg; 9241da177e4SLinus Torvalds long timeo; 9251da177e4SLinus Torvalds 9261da177e4SLinus Torvalds lock_sock(sk); 9271da177e4SLinus Torvalds 9281da177e4SLinus Torvalds flags = msg->msg_flags; 9291da177e4SLinus Torvalds timeo = sock_sndtimeo(sk, flags & MSG_DONTWAIT); 9301da177e4SLinus Torvalds 9311da177e4SLinus Torvalds /* Wait for a connection to finish. */ 9321da177e4SLinus Torvalds if ((1 << sk->sk_state) & ~(TCPF_ESTABLISHED | TCPF_CLOSE_WAIT)) 9331da177e4SLinus Torvalds if ((err = sk_stream_wait_connect(sk, &timeo)) != 0) 9341da177e4SLinus Torvalds goto out_err; 9351da177e4SLinus Torvalds 9361da177e4SLinus Torvalds /* This should be in poll */ 9371da177e4SLinus Torvalds clear_bit(SOCK_ASYNC_NOSPACE, &sk->sk_socket->flags); 9381da177e4SLinus Torvalds 9390c54b85fSIlpo Järvinen mss_now = tcp_send_mss(sk, &size_goal, flags); 9401da177e4SLinus Torvalds 9411da177e4SLinus Torvalds /* Ok commence sending. */ 9421da177e4SLinus Torvalds iovlen = msg->msg_iovlen; 9431da177e4SLinus Torvalds iov = msg->msg_iov; 9441da177e4SLinus Torvalds copied = 0; 9451da177e4SLinus Torvalds 9461da177e4SLinus Torvalds err = -EPIPE; 9471da177e4SLinus Torvalds if (sk->sk_err || (sk->sk_shutdown & SEND_SHUTDOWN)) 9480d6a775eSIlpo Järvinen goto out_err; 9491da177e4SLinus Torvalds 950690e99c4SEric Dumazet sg = !!(sk->sk_route_caps & NETIF_F_SG); 951def87cf4SKrishna Kumar 9521da177e4SLinus Torvalds while (--iovlen >= 0) { 95301db403cSDavid S. Miller size_t seglen = iov->iov_len; 9541da177e4SLinus Torvalds unsigned char __user *from = iov->iov_base; 9551da177e4SLinus Torvalds 9561da177e4SLinus Torvalds iov++; 9571da177e4SLinus Torvalds 9581da177e4SLinus Torvalds while (seglen > 0) { 9596828b92bSHerbert Xu int copy = 0; 9606828b92bSHerbert Xu int max = size_goal; 9611da177e4SLinus Torvalds 962fe067e8aSDavid S. Miller skb = tcp_write_queue_tail(sk); 9636828b92bSHerbert Xu if (tcp_send_head(sk)) { 9646828b92bSHerbert Xu if (skb->ip_summed == CHECKSUM_NONE) 9656828b92bSHerbert Xu max = mss_now; 9666828b92bSHerbert Xu copy = max - skb->len; 9676828b92bSHerbert Xu } 9681da177e4SLinus Torvalds 9696828b92bSHerbert Xu if (copy <= 0) { 9701da177e4SLinus Torvalds new_segment: 9711da177e4SLinus Torvalds /* Allocate new segment. If the interface is SG, 9721da177e4SLinus Torvalds * allocate skb fitting to single page. 9731da177e4SLinus Torvalds */ 9741da177e4SLinus Torvalds if (!sk_stream_memory_free(sk)) 9751da177e4SLinus Torvalds goto wait_for_sndbuf; 9761da177e4SLinus Torvalds 977def87cf4SKrishna Kumar skb = sk_stream_alloc_skb(sk, 978def87cf4SKrishna Kumar select_size(sk, sg), 979df97c708SPavel Emelyanov sk->sk_allocation); 9801da177e4SLinus Torvalds if (!skb) 9811da177e4SLinus Torvalds goto wait_for_memory; 9821da177e4SLinus Torvalds 9831da177e4SLinus Torvalds /* 9841da177e4SLinus Torvalds * Check whether we can use HW checksum. 9851da177e4SLinus Torvalds */ 9868648b305SHerbert Xu if (sk->sk_route_caps & NETIF_F_ALL_CSUM) 98784fa7933SPatrick McHardy skb->ip_summed = CHECKSUM_PARTIAL; 9881da177e4SLinus Torvalds 9899e412ba7SIlpo Järvinen skb_entail(sk, skb); 990c1b4a7e6SDavid S. Miller copy = size_goal; 9916828b92bSHerbert Xu max = size_goal; 9921da177e4SLinus Torvalds } 9931da177e4SLinus Torvalds 9941da177e4SLinus Torvalds /* Try to append data to the end of skb. */ 9951da177e4SLinus Torvalds if (copy > seglen) 9961da177e4SLinus Torvalds copy = seglen; 9971da177e4SLinus Torvalds 9981da177e4SLinus Torvalds /* Where to copy to? */ 999a21d4572SEric Dumazet if (skb_availroom(skb) > 0) { 10001da177e4SLinus Torvalds /* We have some space in skb head. Superb! */ 1001a21d4572SEric Dumazet copy = min_t(int, copy, skb_availroom(skb)); 1002c6e1a0d1STom Herbert err = skb_add_data_nocache(sk, skb, from, copy); 1003c6e1a0d1STom Herbert if (err) 10041da177e4SLinus Torvalds goto do_fault; 10051da177e4SLinus Torvalds } else { 10061da177e4SLinus Torvalds int merge = 0; 10071da177e4SLinus Torvalds int i = skb_shinfo(skb)->nr_frags; 10080a5912dbSEric Dumazet struct page *page = sk->sk_sndmsg_page; 1009761965eaSEric Dumazet int off; 1010761965eaSEric Dumazet 1011761965eaSEric Dumazet if (page && page_count(page) == 1) 10120a5912dbSEric Dumazet sk->sk_sndmsg_off = 0; 1013761965eaSEric Dumazet 10140a5912dbSEric Dumazet off = sk->sk_sndmsg_off; 10151da177e4SLinus Torvalds 10161da177e4SLinus Torvalds if (skb_can_coalesce(skb, i, page, off) && 10171da177e4SLinus Torvalds off != PAGE_SIZE) { 10181da177e4SLinus Torvalds /* We can extend the last page 10191da177e4SLinus Torvalds * fragment. */ 10201da177e4SLinus Torvalds merge = 1; 1021def87cf4SKrishna Kumar } else if (i == MAX_SKB_FRAGS || !sg) { 10221da177e4SLinus Torvalds /* Need to add new fragment and cannot 10231da177e4SLinus Torvalds * do this because interface is non-SG, 10241da177e4SLinus Torvalds * or because all the page slots are 10251da177e4SLinus Torvalds * busy. */ 10261da177e4SLinus Torvalds tcp_mark_push(tp, skb); 10271da177e4SLinus Torvalds goto new_segment; 10281da177e4SLinus Torvalds } else if (page) { 10291da177e4SLinus Torvalds if (off == PAGE_SIZE) { 10301da177e4SLinus Torvalds put_page(page); 10310a5912dbSEric Dumazet sk->sk_sndmsg_page = page = NULL; 1032fb5f5e6eSHerbert Xu off = 0; 10331da177e4SLinus Torvalds } 1034ef015786SHerbert Xu } else 1035fb5f5e6eSHerbert Xu off = 0; 1036ef015786SHerbert Xu 1037ef015786SHerbert Xu if (copy > PAGE_SIZE - off) 1038ef015786SHerbert Xu copy = PAGE_SIZE - off; 1039ef015786SHerbert Xu 10403ab224beSHideo Aoki if (!sk_wmem_schedule(sk, copy)) 1041ef015786SHerbert Xu goto wait_for_memory; 10421da177e4SLinus Torvalds 10431da177e4SLinus Torvalds if (!page) { 10441da177e4SLinus Torvalds /* Allocate new cache page. */ 10451da177e4SLinus Torvalds if (!(page = sk_stream_alloc_page(sk))) 10461da177e4SLinus Torvalds goto wait_for_memory; 10471da177e4SLinus Torvalds } 10481da177e4SLinus Torvalds 10491da177e4SLinus Torvalds /* Time to copy data. We are close to 10501da177e4SLinus Torvalds * the end! */ 1051c6e1a0d1STom Herbert err = skb_copy_to_page_nocache(sk, from, skb, 1052c6e1a0d1STom Herbert page, off, copy); 10531da177e4SLinus Torvalds if (err) { 10541da177e4SLinus Torvalds /* If this page was new, give it to the 10551da177e4SLinus Torvalds * socket so it does not get leaked. 10561da177e4SLinus Torvalds */ 10570a5912dbSEric Dumazet if (!sk->sk_sndmsg_page) { 10580a5912dbSEric Dumazet sk->sk_sndmsg_page = page; 10590a5912dbSEric Dumazet sk->sk_sndmsg_off = 0; 10601da177e4SLinus Torvalds } 10611da177e4SLinus Torvalds goto do_error; 10621da177e4SLinus Torvalds } 10631da177e4SLinus Torvalds 10641da177e4SLinus Torvalds /* Update the skb. */ 10651da177e4SLinus Torvalds if (merge) { 10669e903e08SEric Dumazet skb_frag_size_add(&skb_shinfo(skb)->frags[i - 1], copy); 10671da177e4SLinus Torvalds } else { 10681da177e4SLinus Torvalds skb_fill_page_desc(skb, i, page, off, copy); 10690a5912dbSEric Dumazet if (sk->sk_sndmsg_page) { 10701da177e4SLinus Torvalds get_page(page); 10711da177e4SLinus Torvalds } else if (off + copy < PAGE_SIZE) { 10721da177e4SLinus Torvalds get_page(page); 10730a5912dbSEric Dumazet sk->sk_sndmsg_page = page; 10741da177e4SLinus Torvalds } 10751da177e4SLinus Torvalds } 10761da177e4SLinus Torvalds 10770a5912dbSEric Dumazet sk->sk_sndmsg_off = off + copy; 10781da177e4SLinus Torvalds } 10791da177e4SLinus Torvalds 10801da177e4SLinus Torvalds if (!copied) 10814de075e0SEric Dumazet TCP_SKB_CB(skb)->tcp_flags &= ~TCPHDR_PSH; 10821da177e4SLinus Torvalds 10831da177e4SLinus Torvalds tp->write_seq += copy; 10841da177e4SLinus Torvalds TCP_SKB_CB(skb)->end_seq += copy; 10857967168cSHerbert Xu skb_shinfo(skb)->gso_segs = 0; 10861da177e4SLinus Torvalds 10871da177e4SLinus Torvalds from += copy; 10881da177e4SLinus Torvalds copied += copy; 10891da177e4SLinus Torvalds if ((seglen -= copy) == 0 && iovlen == 0) 10901da177e4SLinus Torvalds goto out; 10911da177e4SLinus Torvalds 10926828b92bSHerbert Xu if (skb->len < max || (flags & MSG_OOB)) 10931da177e4SLinus Torvalds continue; 10941da177e4SLinus Torvalds 10951da177e4SLinus Torvalds if (forced_push(tp)) { 10961da177e4SLinus Torvalds tcp_mark_push(tp, skb); 10979e412ba7SIlpo Järvinen __tcp_push_pending_frames(sk, mss_now, TCP_NAGLE_PUSH); 1098fe067e8aSDavid S. Miller } else if (skb == tcp_send_head(sk)) 10991da177e4SLinus Torvalds tcp_push_one(sk, mss_now); 11001da177e4SLinus Torvalds continue; 11011da177e4SLinus Torvalds 11021da177e4SLinus Torvalds wait_for_sndbuf: 11031da177e4SLinus Torvalds set_bit(SOCK_NOSPACE, &sk->sk_socket->flags); 11041da177e4SLinus Torvalds wait_for_memory: 11051da177e4SLinus Torvalds if (copied) 11069e412ba7SIlpo Järvinen tcp_push(sk, flags & ~MSG_MORE, mss_now, TCP_NAGLE_PUSH); 11071da177e4SLinus Torvalds 11081da177e4SLinus Torvalds if ((err = sk_stream_wait_memory(sk, &timeo)) != 0) 11091da177e4SLinus Torvalds goto do_error; 11101da177e4SLinus Torvalds 11110c54b85fSIlpo Järvinen mss_now = tcp_send_mss(sk, &size_goal, flags); 11121da177e4SLinus Torvalds } 11131da177e4SLinus Torvalds } 11141da177e4SLinus Torvalds 11151da177e4SLinus Torvalds out: 11161da177e4SLinus Torvalds if (copied) 11179e412ba7SIlpo Järvinen tcp_push(sk, flags, mss_now, tp->nonagle); 11181da177e4SLinus Torvalds release_sock(sk); 11191da177e4SLinus Torvalds return copied; 11201da177e4SLinus Torvalds 11211da177e4SLinus Torvalds do_fault: 11221da177e4SLinus Torvalds if (!skb->len) { 1123fe067e8aSDavid S. Miller tcp_unlink_write_queue(skb, sk); 1124fe067e8aSDavid S. Miller /* It is the one place in all of TCP, except connection 1125fe067e8aSDavid S. Miller * reset, where we can be unlinking the send_head. 1126fe067e8aSDavid S. Miller */ 1127fe067e8aSDavid S. Miller tcp_check_send_head(sk, skb); 11283ab224beSHideo Aoki sk_wmem_free_skb(sk, skb); 11291da177e4SLinus Torvalds } 11301da177e4SLinus Torvalds 11311da177e4SLinus Torvalds do_error: 11321da177e4SLinus Torvalds if (copied) 11331da177e4SLinus Torvalds goto out; 11341da177e4SLinus Torvalds out_err: 11351da177e4SLinus Torvalds err = sk_stream_error(sk, flags, err); 11361da177e4SLinus Torvalds release_sock(sk); 11371da177e4SLinus Torvalds return err; 11381da177e4SLinus Torvalds } 11394bc2f18bSEric Dumazet EXPORT_SYMBOL(tcp_sendmsg); 11401da177e4SLinus Torvalds 11411da177e4SLinus Torvalds /* 11421da177e4SLinus Torvalds * Handle reading urgent data. BSD has very simple semantics for 11431da177e4SLinus Torvalds * this, no blocking and very strange errors 8) 11441da177e4SLinus Torvalds */ 11451da177e4SLinus Torvalds 1146377f0a08SRami Rosen static int tcp_recv_urg(struct sock *sk, struct msghdr *msg, int len, int flags) 11471da177e4SLinus Torvalds { 11481da177e4SLinus Torvalds struct tcp_sock *tp = tcp_sk(sk); 11491da177e4SLinus Torvalds 11501da177e4SLinus Torvalds /* No URG data to read. */ 11511da177e4SLinus Torvalds if (sock_flag(sk, SOCK_URGINLINE) || !tp->urg_data || 11521da177e4SLinus Torvalds tp->urg_data == TCP_URG_READ) 11531da177e4SLinus Torvalds return -EINVAL; /* Yes this is right ! */ 11541da177e4SLinus Torvalds 11551da177e4SLinus Torvalds if (sk->sk_state == TCP_CLOSE && !sock_flag(sk, SOCK_DONE)) 11561da177e4SLinus Torvalds return -ENOTCONN; 11571da177e4SLinus Torvalds 11581da177e4SLinus Torvalds if (tp->urg_data & TCP_URG_VALID) { 11591da177e4SLinus Torvalds int err = 0; 11601da177e4SLinus Torvalds char c = tp->urg_data; 11611da177e4SLinus Torvalds 11621da177e4SLinus Torvalds if (!(flags & MSG_PEEK)) 11631da177e4SLinus Torvalds tp->urg_data = TCP_URG_READ; 11641da177e4SLinus Torvalds 11651da177e4SLinus Torvalds /* Read urgent data. */ 11661da177e4SLinus Torvalds msg->msg_flags |= MSG_OOB; 11671da177e4SLinus Torvalds 11681da177e4SLinus Torvalds if (len > 0) { 11691da177e4SLinus Torvalds if (!(flags & MSG_TRUNC)) 11701da177e4SLinus Torvalds err = memcpy_toiovec(msg->msg_iov, &c, 1); 11711da177e4SLinus Torvalds len = 1; 11721da177e4SLinus Torvalds } else 11731da177e4SLinus Torvalds msg->msg_flags |= MSG_TRUNC; 11741da177e4SLinus Torvalds 11751da177e4SLinus Torvalds return err ? -EFAULT : len; 11761da177e4SLinus Torvalds } 11771da177e4SLinus Torvalds 11781da177e4SLinus Torvalds if (sk->sk_state == TCP_CLOSE || (sk->sk_shutdown & RCV_SHUTDOWN)) 11791da177e4SLinus Torvalds return 0; 11801da177e4SLinus Torvalds 11811da177e4SLinus Torvalds /* Fixed the recv(..., MSG_OOB) behaviour. BSD docs and 11821da177e4SLinus Torvalds * the available implementations agree in this case: 11831da177e4SLinus Torvalds * this call should never block, independent of the 11841da177e4SLinus Torvalds * blocking state of the socket. 11851da177e4SLinus Torvalds * Mike <pall@rz.uni-karlsruhe.de> 11861da177e4SLinus Torvalds */ 11871da177e4SLinus Torvalds return -EAGAIN; 11881da177e4SLinus Torvalds } 11891da177e4SLinus Torvalds 11901da177e4SLinus Torvalds /* Clean up the receive buffer for full frames taken by the user, 11911da177e4SLinus Torvalds * then send an ACK if necessary. COPIED is the number of bytes 11921da177e4SLinus Torvalds * tcp_recvmsg has given to the user so far, it speeds up the 11931da177e4SLinus Torvalds * calculation of whether or not we must ACK for the sake of 11941da177e4SLinus Torvalds * a window update. 11951da177e4SLinus Torvalds */ 11960e4b4992SChris Leech void tcp_cleanup_rbuf(struct sock *sk, int copied) 11971da177e4SLinus Torvalds { 11981da177e4SLinus Torvalds struct tcp_sock *tp = tcp_sk(sk); 11991da177e4SLinus Torvalds int time_to_ack = 0; 12001da177e4SLinus Torvalds 12011da177e4SLinus Torvalds struct sk_buff *skb = skb_peek(&sk->sk_receive_queue); 12021da177e4SLinus Torvalds 1203d792c100SIlpo Järvinen WARN(skb && !before(tp->copied_seq, TCP_SKB_CB(skb)->end_seq), 12042af6fd8bSJoe Perches "cleanup rbuf bug: copied %X seq %X rcvnxt %X\n", 1205d792c100SIlpo Järvinen tp->copied_seq, TCP_SKB_CB(skb)->end_seq, tp->rcv_nxt); 12061da177e4SLinus Torvalds 1207463c84b9SArnaldo Carvalho de Melo if (inet_csk_ack_scheduled(sk)) { 1208463c84b9SArnaldo Carvalho de Melo const struct inet_connection_sock *icsk = inet_csk(sk); 12091da177e4SLinus Torvalds /* Delayed ACKs frequently hit locked sockets during bulk 12101da177e4SLinus Torvalds * receive. */ 1211463c84b9SArnaldo Carvalho de Melo if (icsk->icsk_ack.blocked || 12121da177e4SLinus Torvalds /* Once-per-two-segments ACK was not sent by tcp_input.c */ 1213463c84b9SArnaldo Carvalho de Melo tp->rcv_nxt - tp->rcv_wup > icsk->icsk_ack.rcv_mss || 12141da177e4SLinus Torvalds /* 12151da177e4SLinus Torvalds * If this read emptied read buffer, we send ACK, if 12161da177e4SLinus Torvalds * connection is not bidirectional, user drained 12171da177e4SLinus Torvalds * receive buffer and there was a small segment 12181da177e4SLinus Torvalds * in queue. 12191da177e4SLinus Torvalds */ 12201ef9696cSAlexey Kuznetsov (copied > 0 && 12211ef9696cSAlexey Kuznetsov ((icsk->icsk_ack.pending & ICSK_ACK_PUSHED2) || 12221ef9696cSAlexey Kuznetsov ((icsk->icsk_ack.pending & ICSK_ACK_PUSHED) && 12231ef9696cSAlexey Kuznetsov !icsk->icsk_ack.pingpong)) && 12241ef9696cSAlexey Kuznetsov !atomic_read(&sk->sk_rmem_alloc))) 12251da177e4SLinus Torvalds time_to_ack = 1; 12261da177e4SLinus Torvalds } 12271da177e4SLinus Torvalds 12281da177e4SLinus Torvalds /* We send an ACK if we can now advertise a non-zero window 12291da177e4SLinus Torvalds * which has been raised "significantly". 12301da177e4SLinus Torvalds * 12311da177e4SLinus Torvalds * Even if window raised up to infinity, do not send window open ACK 12321da177e4SLinus Torvalds * in states, where we will not receive more. It is useless. 12331da177e4SLinus Torvalds */ 12341da177e4SLinus Torvalds if (copied > 0 && !time_to_ack && !(sk->sk_shutdown & RCV_SHUTDOWN)) { 12351da177e4SLinus Torvalds __u32 rcv_window_now = tcp_receive_window(tp); 12361da177e4SLinus Torvalds 12371da177e4SLinus Torvalds /* Optimize, __tcp_select_window() is not cheap. */ 12381da177e4SLinus Torvalds if (2*rcv_window_now <= tp->window_clamp) { 12391da177e4SLinus Torvalds __u32 new_window = __tcp_select_window(sk); 12401da177e4SLinus Torvalds 12411da177e4SLinus Torvalds /* Send ACK now, if this read freed lots of space 12421da177e4SLinus Torvalds * in our buffer. Certainly, new_window is new window. 12431da177e4SLinus Torvalds * We can advertise it now, if it is not less than current one. 12441da177e4SLinus Torvalds * "Lots" means "at least twice" here. 12451da177e4SLinus Torvalds */ 12461da177e4SLinus Torvalds if (new_window && new_window >= 2 * rcv_window_now) 12471da177e4SLinus Torvalds time_to_ack = 1; 12481da177e4SLinus Torvalds } 12491da177e4SLinus Torvalds } 12501da177e4SLinus Torvalds if (time_to_ack) 12511da177e4SLinus Torvalds tcp_send_ack(sk); 12521da177e4SLinus Torvalds } 12531da177e4SLinus Torvalds 12541da177e4SLinus Torvalds static void tcp_prequeue_process(struct sock *sk) 12551da177e4SLinus Torvalds { 12561da177e4SLinus Torvalds struct sk_buff *skb; 12571da177e4SLinus Torvalds struct tcp_sock *tp = tcp_sk(sk); 12581da177e4SLinus Torvalds 12596f67c817SPavel Emelyanov NET_INC_STATS_USER(sock_net(sk), LINUX_MIB_TCPPREQUEUED); 12601da177e4SLinus Torvalds 12611da177e4SLinus Torvalds /* RX process wants to run with disabled BHs, though it is not 12621da177e4SLinus Torvalds * necessary */ 12631da177e4SLinus Torvalds local_bh_disable(); 12641da177e4SLinus Torvalds while ((skb = __skb_dequeue(&tp->ucopy.prequeue)) != NULL) 1265c57943a1SPeter Zijlstra sk_backlog_rcv(sk, skb); 12661da177e4SLinus Torvalds local_bh_enable(); 12671da177e4SLinus Torvalds 12681da177e4SLinus Torvalds /* Clear memory counter. */ 12691da177e4SLinus Torvalds tp->ucopy.memory = 0; 12701da177e4SLinus Torvalds } 12711da177e4SLinus Torvalds 127273852e81SSteven J. Magnani #ifdef CONFIG_NET_DMA 127373852e81SSteven J. Magnani static void tcp_service_net_dma(struct sock *sk, bool wait) 127473852e81SSteven J. Magnani { 127573852e81SSteven J. Magnani dma_cookie_t done, used; 127673852e81SSteven J. Magnani dma_cookie_t last_issued; 127773852e81SSteven J. Magnani struct tcp_sock *tp = tcp_sk(sk); 127873852e81SSteven J. Magnani 127973852e81SSteven J. Magnani if (!tp->ucopy.dma_chan) 128073852e81SSteven J. Magnani return; 128173852e81SSteven J. Magnani 128273852e81SSteven J. Magnani last_issued = tp->ucopy.dma_cookie; 128373852e81SSteven J. Magnani dma_async_memcpy_issue_pending(tp->ucopy.dma_chan); 128473852e81SSteven J. Magnani 128573852e81SSteven J. Magnani do { 128673852e81SSteven J. Magnani if (dma_async_memcpy_complete(tp->ucopy.dma_chan, 128773852e81SSteven J. Magnani last_issued, &done, 128873852e81SSteven J. Magnani &used) == DMA_SUCCESS) { 128973852e81SSteven J. Magnani /* Safe to free early-copied skbs now */ 129073852e81SSteven J. Magnani __skb_queue_purge(&sk->sk_async_wait_queue); 129173852e81SSteven J. Magnani break; 129273852e81SSteven J. Magnani } else { 129373852e81SSteven J. Magnani struct sk_buff *skb; 129473852e81SSteven J. Magnani while ((skb = skb_peek(&sk->sk_async_wait_queue)) && 129573852e81SSteven J. Magnani (dma_async_is_complete(skb->dma_cookie, done, 129673852e81SSteven J. Magnani used) == DMA_SUCCESS)) { 129773852e81SSteven J. Magnani __skb_dequeue(&sk->sk_async_wait_queue); 129873852e81SSteven J. Magnani kfree_skb(skb); 129973852e81SSteven J. Magnani } 130073852e81SSteven J. Magnani } 130173852e81SSteven J. Magnani } while (wait); 130273852e81SSteven J. Magnani } 130373852e81SSteven J. Magnani #endif 130473852e81SSteven J. Magnani 13051da177e4SLinus Torvalds static inline struct sk_buff *tcp_recv_skb(struct sock *sk, u32 seq, u32 *off) 13061da177e4SLinus Torvalds { 13071da177e4SLinus Torvalds struct sk_buff *skb; 13081da177e4SLinus Torvalds u32 offset; 13091da177e4SLinus Torvalds 13101da177e4SLinus Torvalds skb_queue_walk(&sk->sk_receive_queue, skb) { 13111da177e4SLinus Torvalds offset = seq - TCP_SKB_CB(skb)->seq; 1312aa8223c7SArnaldo Carvalho de Melo if (tcp_hdr(skb)->syn) 13131da177e4SLinus Torvalds offset--; 1314aa8223c7SArnaldo Carvalho de Melo if (offset < skb->len || tcp_hdr(skb)->fin) { 13151da177e4SLinus Torvalds *off = offset; 13161da177e4SLinus Torvalds return skb; 13171da177e4SLinus Torvalds } 13181da177e4SLinus Torvalds } 13191da177e4SLinus Torvalds return NULL; 13201da177e4SLinus Torvalds } 13211da177e4SLinus Torvalds 13221da177e4SLinus Torvalds /* 13231da177e4SLinus Torvalds * This routine provides an alternative to tcp_recvmsg() for routines 13241da177e4SLinus Torvalds * that would like to handle copying from skbuffs directly in 'sendfile' 13251da177e4SLinus Torvalds * fashion. 13261da177e4SLinus Torvalds * Note: 13271da177e4SLinus Torvalds * - It is assumed that the socket was locked by the caller. 13281da177e4SLinus Torvalds * - The routine does not block. 13291da177e4SLinus Torvalds * - At present, there is no support for reading OOB data 13301da177e4SLinus Torvalds * or for 'peeking' the socket using this routine 13311da177e4SLinus Torvalds * (although both would be easy to implement). 13321da177e4SLinus Torvalds */ 13331da177e4SLinus Torvalds int tcp_read_sock(struct sock *sk, read_descriptor_t *desc, 13341da177e4SLinus Torvalds sk_read_actor_t recv_actor) 13351da177e4SLinus Torvalds { 13361da177e4SLinus Torvalds struct sk_buff *skb; 13371da177e4SLinus Torvalds struct tcp_sock *tp = tcp_sk(sk); 13381da177e4SLinus Torvalds u32 seq = tp->copied_seq; 13391da177e4SLinus Torvalds u32 offset; 13401da177e4SLinus Torvalds int copied = 0; 13411da177e4SLinus Torvalds 13421da177e4SLinus Torvalds if (sk->sk_state == TCP_LISTEN) 13431da177e4SLinus Torvalds return -ENOTCONN; 13441da177e4SLinus Torvalds while ((skb = tcp_recv_skb(sk, seq, &offset)) != NULL) { 13451da177e4SLinus Torvalds if (offset < skb->len) { 1346374e7b59SOctavian Purdila int used; 1347374e7b59SOctavian Purdila size_t len; 13481da177e4SLinus Torvalds 13491da177e4SLinus Torvalds len = skb->len - offset; 13501da177e4SLinus Torvalds /* Stop reading if we hit a patch of urgent data */ 13511da177e4SLinus Torvalds if (tp->urg_data) { 13521da177e4SLinus Torvalds u32 urg_offset = tp->urg_seq - seq; 13531da177e4SLinus Torvalds if (urg_offset < len) 13541da177e4SLinus Torvalds len = urg_offset; 13551da177e4SLinus Torvalds if (!len) 13561da177e4SLinus Torvalds break; 13571da177e4SLinus Torvalds } 13581da177e4SLinus Torvalds used = recv_actor(desc, skb, offset, len); 1359ddb61a57SJens Axboe if (used < 0) { 1360ddb61a57SJens Axboe if (!copied) 1361ddb61a57SJens Axboe copied = used; 1362ddb61a57SJens Axboe break; 1363ddb61a57SJens Axboe } else if (used <= len) { 13641da177e4SLinus Torvalds seq += used; 13651da177e4SLinus Torvalds copied += used; 13661da177e4SLinus Torvalds offset += used; 13671da177e4SLinus Torvalds } 1368293ad604SOctavian Purdila /* 1369293ad604SOctavian Purdila * If recv_actor drops the lock (e.g. TCP splice 1370293ad604SOctavian Purdila * receive) the skb pointer might be invalid when 1371293ad604SOctavian Purdila * getting here: tcp_collapse might have deleted it 1372293ad604SOctavian Purdila * while aggregating skbs from the socket queue. 1373293ad604SOctavian Purdila */ 1374293ad604SOctavian Purdila skb = tcp_recv_skb(sk, seq-1, &offset); 1375293ad604SOctavian Purdila if (!skb || (offset+1 != skb->len)) 13761da177e4SLinus Torvalds break; 13771da177e4SLinus Torvalds } 1378aa8223c7SArnaldo Carvalho de Melo if (tcp_hdr(skb)->fin) { 1379624d1164SChris Leech sk_eat_skb(sk, skb, 0); 13801da177e4SLinus Torvalds ++seq; 13811da177e4SLinus Torvalds break; 13821da177e4SLinus Torvalds } 1383624d1164SChris Leech sk_eat_skb(sk, skb, 0); 13841da177e4SLinus Torvalds if (!desc->count) 13851da177e4SLinus Torvalds break; 1386baff42abSSteven J. Magnani tp->copied_seq = seq; 13871da177e4SLinus Torvalds } 13881da177e4SLinus Torvalds tp->copied_seq = seq; 13891da177e4SLinus Torvalds 13901da177e4SLinus Torvalds tcp_rcv_space_adjust(sk); 13911da177e4SLinus Torvalds 13921da177e4SLinus Torvalds /* Clean up data we have read: This will do ACK frames. */ 1393ddb61a57SJens Axboe if (copied > 0) 13940e4b4992SChris Leech tcp_cleanup_rbuf(sk, copied); 13951da177e4SLinus Torvalds return copied; 13961da177e4SLinus Torvalds } 13974bc2f18bSEric Dumazet EXPORT_SYMBOL(tcp_read_sock); 13981da177e4SLinus Torvalds 13991da177e4SLinus Torvalds /* 14001da177e4SLinus Torvalds * This routine copies from a sock struct into the user buffer. 14011da177e4SLinus Torvalds * 14021da177e4SLinus Torvalds * Technical note: in 2.3 we work on _locked_ socket, so that 14031da177e4SLinus Torvalds * tricks with *seq access order and skb->users are not required. 14041da177e4SLinus Torvalds * Probably, code can be easily improved even more. 14051da177e4SLinus Torvalds */ 14061da177e4SLinus Torvalds 14071da177e4SLinus Torvalds int tcp_recvmsg(struct kiocb *iocb, struct sock *sk, struct msghdr *msg, 14081da177e4SLinus Torvalds size_t len, int nonblock, int flags, int *addr_len) 14091da177e4SLinus Torvalds { 14101da177e4SLinus Torvalds struct tcp_sock *tp = tcp_sk(sk); 14111da177e4SLinus Torvalds int copied = 0; 14121da177e4SLinus Torvalds u32 peek_seq; 14131da177e4SLinus Torvalds u32 *seq; 14141da177e4SLinus Torvalds unsigned long used; 14151da177e4SLinus Torvalds int err; 14161da177e4SLinus Torvalds int target; /* Read at least this many bytes */ 14171da177e4SLinus Torvalds long timeo; 14181da177e4SLinus Torvalds struct task_struct *user_recv = NULL; 14191a2449a8SChris Leech int copied_early = 0; 14202b1244a4SChris Leech struct sk_buff *skb; 142177527313SIlpo Järvinen u32 urg_hole = 0; 14221da177e4SLinus Torvalds 14231da177e4SLinus Torvalds lock_sock(sk); 14241da177e4SLinus Torvalds 14251da177e4SLinus Torvalds err = -ENOTCONN; 14261da177e4SLinus Torvalds if (sk->sk_state == TCP_LISTEN) 14271da177e4SLinus Torvalds goto out; 14281da177e4SLinus Torvalds 14291da177e4SLinus Torvalds timeo = sock_rcvtimeo(sk, nonblock); 14301da177e4SLinus Torvalds 14311da177e4SLinus Torvalds /* Urgent data needs to be handled specially. */ 14321da177e4SLinus Torvalds if (flags & MSG_OOB) 14331da177e4SLinus Torvalds goto recv_urg; 14341da177e4SLinus Torvalds 14351da177e4SLinus Torvalds seq = &tp->copied_seq; 14361da177e4SLinus Torvalds if (flags & MSG_PEEK) { 14371da177e4SLinus Torvalds peek_seq = tp->copied_seq; 14381da177e4SLinus Torvalds seq = &peek_seq; 14391da177e4SLinus Torvalds } 14401da177e4SLinus Torvalds 14411da177e4SLinus Torvalds target = sock_rcvlowat(sk, flags & MSG_WAITALL, len); 14421da177e4SLinus Torvalds 14431a2449a8SChris Leech #ifdef CONFIG_NET_DMA 14441a2449a8SChris Leech tp->ucopy.dma_chan = NULL; 14451a2449a8SChris Leech preempt_disable(); 14462b1244a4SChris Leech skb = skb_peek_tail(&sk->sk_receive_queue); 1447e00c5d8bSAndrew Morton { 1448e00c5d8bSAndrew Morton int available = 0; 1449e00c5d8bSAndrew Morton 14502b1244a4SChris Leech if (skb) 14512b1244a4SChris Leech available = TCP_SKB_CB(skb)->seq + skb->len - (*seq); 14522b1244a4SChris Leech if ((available < target) && 14532b1244a4SChris Leech (len > sysctl_tcp_dma_copybreak) && !(flags & MSG_PEEK) && 1454e00c5d8bSAndrew Morton !sysctl_tcp_low_latency && 1455a2bd1140SDave Jiang net_dma_find_channel()) { 14561a2449a8SChris Leech preempt_enable_no_resched(); 1457e00c5d8bSAndrew Morton tp->ucopy.pinned_list = 1458e00c5d8bSAndrew Morton dma_pin_iovec_pages(msg->msg_iov, len); 1459e00c5d8bSAndrew Morton } else { 14601a2449a8SChris Leech preempt_enable_no_resched(); 1461e00c5d8bSAndrew Morton } 1462e00c5d8bSAndrew Morton } 14631a2449a8SChris Leech #endif 14641a2449a8SChris Leech 14651da177e4SLinus Torvalds do { 14661da177e4SLinus Torvalds u32 offset; 14671da177e4SLinus Torvalds 14681da177e4SLinus Torvalds /* Are we at urgent data? Stop if we have read anything or have SIGURG pending. */ 14691da177e4SLinus Torvalds if (tp->urg_data && tp->urg_seq == *seq) { 14701da177e4SLinus Torvalds if (copied) 14711da177e4SLinus Torvalds break; 14721da177e4SLinus Torvalds if (signal_pending(current)) { 14731da177e4SLinus Torvalds copied = timeo ? sock_intr_errno(timeo) : -EAGAIN; 14741da177e4SLinus Torvalds break; 14751da177e4SLinus Torvalds } 14761da177e4SLinus Torvalds } 14771da177e4SLinus Torvalds 14781da177e4SLinus Torvalds /* Next get a buffer. */ 14791da177e4SLinus Torvalds 148091521944SDavid S. Miller skb_queue_walk(&sk->sk_receive_queue, skb) { 14811da177e4SLinus Torvalds /* Now that we have two receive queues this 14821da177e4SLinus Torvalds * shouldn't happen. 14831da177e4SLinus Torvalds */ 1484d792c100SIlpo Järvinen if (WARN(before(*seq, TCP_SKB_CB(skb)->seq), 14852af6fd8bSJoe Perches "recvmsg bug: copied %X seq %X rcvnxt %X fl %X\n", 14862af6fd8bSJoe Perches *seq, TCP_SKB_CB(skb)->seq, tp->rcv_nxt, 1487d792c100SIlpo Järvinen flags)) 14881da177e4SLinus Torvalds break; 1489d792c100SIlpo Järvinen 14901da177e4SLinus Torvalds offset = *seq - TCP_SKB_CB(skb)->seq; 1491aa8223c7SArnaldo Carvalho de Melo if (tcp_hdr(skb)->syn) 14921da177e4SLinus Torvalds offset--; 14931da177e4SLinus Torvalds if (offset < skb->len) 14941da177e4SLinus Torvalds goto found_ok_skb; 1495aa8223c7SArnaldo Carvalho de Melo if (tcp_hdr(skb)->fin) 14961da177e4SLinus Torvalds goto found_fin_ok; 14972af6fd8bSJoe Perches WARN(!(flags & MSG_PEEK), 14982af6fd8bSJoe Perches "recvmsg bug 2: copied %X seq %X rcvnxt %X fl %X\n", 14992af6fd8bSJoe Perches *seq, TCP_SKB_CB(skb)->seq, tp->rcv_nxt, flags); 150091521944SDavid S. Miller } 15011da177e4SLinus Torvalds 15021da177e4SLinus Torvalds /* Well, if we have backlog, try to process it now yet. */ 15031da177e4SLinus Torvalds 15041da177e4SLinus Torvalds if (copied >= target && !sk->sk_backlog.tail) 15051da177e4SLinus Torvalds break; 15061da177e4SLinus Torvalds 15071da177e4SLinus Torvalds if (copied) { 15081da177e4SLinus Torvalds if (sk->sk_err || 15091da177e4SLinus Torvalds sk->sk_state == TCP_CLOSE || 15101da177e4SLinus Torvalds (sk->sk_shutdown & RCV_SHUTDOWN) || 15111da177e4SLinus Torvalds !timeo || 1512518a09efSDavid S. Miller signal_pending(current)) 15131da177e4SLinus Torvalds break; 15141da177e4SLinus Torvalds } else { 15151da177e4SLinus Torvalds if (sock_flag(sk, SOCK_DONE)) 15161da177e4SLinus Torvalds break; 15171da177e4SLinus Torvalds 15181da177e4SLinus Torvalds if (sk->sk_err) { 15191da177e4SLinus Torvalds copied = sock_error(sk); 15201da177e4SLinus Torvalds break; 15211da177e4SLinus Torvalds } 15221da177e4SLinus Torvalds 15231da177e4SLinus Torvalds if (sk->sk_shutdown & RCV_SHUTDOWN) 15241da177e4SLinus Torvalds break; 15251da177e4SLinus Torvalds 15261da177e4SLinus Torvalds if (sk->sk_state == TCP_CLOSE) { 15271da177e4SLinus Torvalds if (!sock_flag(sk, SOCK_DONE)) { 15281da177e4SLinus Torvalds /* This occurs when user tries to read 15291da177e4SLinus Torvalds * from never connected socket. 15301da177e4SLinus Torvalds */ 15311da177e4SLinus Torvalds copied = -ENOTCONN; 15321da177e4SLinus Torvalds break; 15331da177e4SLinus Torvalds } 15341da177e4SLinus Torvalds break; 15351da177e4SLinus Torvalds } 15361da177e4SLinus Torvalds 15371da177e4SLinus Torvalds if (!timeo) { 15381da177e4SLinus Torvalds copied = -EAGAIN; 15391da177e4SLinus Torvalds break; 15401da177e4SLinus Torvalds } 15411da177e4SLinus Torvalds 15421da177e4SLinus Torvalds if (signal_pending(current)) { 15431da177e4SLinus Torvalds copied = sock_intr_errno(timeo); 15441da177e4SLinus Torvalds break; 15451da177e4SLinus Torvalds } 15461da177e4SLinus Torvalds } 15471da177e4SLinus Torvalds 15480e4b4992SChris Leech tcp_cleanup_rbuf(sk, copied); 15491da177e4SLinus Torvalds 15507df55125SDavid S. Miller if (!sysctl_tcp_low_latency && tp->ucopy.task == user_recv) { 15511da177e4SLinus Torvalds /* Install new reader */ 15521da177e4SLinus Torvalds if (!user_recv && !(flags & (MSG_TRUNC | MSG_PEEK))) { 15531da177e4SLinus Torvalds user_recv = current; 15541da177e4SLinus Torvalds tp->ucopy.task = user_recv; 15551da177e4SLinus Torvalds tp->ucopy.iov = msg->msg_iov; 15561da177e4SLinus Torvalds } 15571da177e4SLinus Torvalds 15581da177e4SLinus Torvalds tp->ucopy.len = len; 15591da177e4SLinus Torvalds 1560547b792cSIlpo Järvinen WARN_ON(tp->copied_seq != tp->rcv_nxt && 1561547b792cSIlpo Järvinen !(flags & (MSG_PEEK | MSG_TRUNC))); 15621da177e4SLinus Torvalds 15631da177e4SLinus Torvalds /* Ugly... If prequeue is not empty, we have to 15641da177e4SLinus Torvalds * process it before releasing socket, otherwise 15651da177e4SLinus Torvalds * order will be broken at second iteration. 15661da177e4SLinus Torvalds * More elegant solution is required!!! 15671da177e4SLinus Torvalds * 15681da177e4SLinus Torvalds * Look: we have the following (pseudo)queues: 15691da177e4SLinus Torvalds * 15701da177e4SLinus Torvalds * 1. packets in flight 15711da177e4SLinus Torvalds * 2. backlog 15721da177e4SLinus Torvalds * 3. prequeue 15731da177e4SLinus Torvalds * 4. receive_queue 15741da177e4SLinus Torvalds * 15751da177e4SLinus Torvalds * Each queue can be processed only if the next ones 15761da177e4SLinus Torvalds * are empty. At this point we have empty receive_queue. 15771da177e4SLinus Torvalds * But prequeue _can_ be not empty after 2nd iteration, 15781da177e4SLinus Torvalds * when we jumped to start of loop because backlog 15791da177e4SLinus Torvalds * processing added something to receive_queue. 15801da177e4SLinus Torvalds * We cannot release_sock(), because backlog contains 15811da177e4SLinus Torvalds * packets arrived _after_ prequeued ones. 15821da177e4SLinus Torvalds * 15831da177e4SLinus Torvalds * Shortly, algorithm is clear --- to process all 15841da177e4SLinus Torvalds * the queues in order. We could make it more directly, 15851da177e4SLinus Torvalds * requeueing packets from backlog to prequeue, if 15861da177e4SLinus Torvalds * is not empty. It is more elegant, but eats cycles, 15871da177e4SLinus Torvalds * unfortunately. 15881da177e4SLinus Torvalds */ 1589b03efcfbSDavid S. Miller if (!skb_queue_empty(&tp->ucopy.prequeue)) 15901da177e4SLinus Torvalds goto do_prequeue; 15911da177e4SLinus Torvalds 15921da177e4SLinus Torvalds /* __ Set realtime policy in scheduler __ */ 15931da177e4SLinus Torvalds } 15941da177e4SLinus Torvalds 159573852e81SSteven J. Magnani #ifdef CONFIG_NET_DMA 159673852e81SSteven J. Magnani if (tp->ucopy.dma_chan) 159773852e81SSteven J. Magnani dma_async_memcpy_issue_pending(tp->ucopy.dma_chan); 159873852e81SSteven J. Magnani #endif 15991da177e4SLinus Torvalds if (copied >= target) { 16001da177e4SLinus Torvalds /* Do not sleep, just process backlog. */ 16011da177e4SLinus Torvalds release_sock(sk); 16021da177e4SLinus Torvalds lock_sock(sk); 16031da177e4SLinus Torvalds } else 16041da177e4SLinus Torvalds sk_wait_data(sk, &timeo); 16051da177e4SLinus Torvalds 16061a2449a8SChris Leech #ifdef CONFIG_NET_DMA 160773852e81SSteven J. Magnani tcp_service_net_dma(sk, false); /* Don't block */ 16081a2449a8SChris Leech tp->ucopy.wakeup = 0; 16091a2449a8SChris Leech #endif 16101a2449a8SChris Leech 16111da177e4SLinus Torvalds if (user_recv) { 16121da177e4SLinus Torvalds int chunk; 16131da177e4SLinus Torvalds 16141da177e4SLinus Torvalds /* __ Restore normal policy in scheduler __ */ 16151da177e4SLinus Torvalds 16161da177e4SLinus Torvalds if ((chunk = len - tp->ucopy.len) != 0) { 1617ed88098eSPavel Emelyanov NET_ADD_STATS_USER(sock_net(sk), LINUX_MIB_TCPDIRECTCOPYFROMBACKLOG, chunk); 16181da177e4SLinus Torvalds len -= chunk; 16191da177e4SLinus Torvalds copied += chunk; 16201da177e4SLinus Torvalds } 16211da177e4SLinus Torvalds 16221da177e4SLinus Torvalds if (tp->rcv_nxt == tp->copied_seq && 1623b03efcfbSDavid S. Miller !skb_queue_empty(&tp->ucopy.prequeue)) { 16241da177e4SLinus Torvalds do_prequeue: 16251da177e4SLinus Torvalds tcp_prequeue_process(sk); 16261da177e4SLinus Torvalds 16271da177e4SLinus Torvalds if ((chunk = len - tp->ucopy.len) != 0) { 1628ed88098eSPavel Emelyanov NET_ADD_STATS_USER(sock_net(sk), LINUX_MIB_TCPDIRECTCOPYFROMPREQUEUE, chunk); 16291da177e4SLinus Torvalds len -= chunk; 16301da177e4SLinus Torvalds copied += chunk; 16311da177e4SLinus Torvalds } 16321da177e4SLinus Torvalds } 16331da177e4SLinus Torvalds } 163477527313SIlpo Järvinen if ((flags & MSG_PEEK) && 163577527313SIlpo Järvinen (peek_seq - copied - urg_hole != tp->copied_seq)) { 16361da177e4SLinus Torvalds if (net_ratelimit()) 16371da177e4SLinus Torvalds printk(KERN_DEBUG "TCP(%s:%d): Application bug, race in MSG_PEEK.\n", 1638ba25f9dcSPavel Emelyanov current->comm, task_pid_nr(current)); 16391da177e4SLinus Torvalds peek_seq = tp->copied_seq; 16401da177e4SLinus Torvalds } 16411da177e4SLinus Torvalds continue; 16421da177e4SLinus Torvalds 16431da177e4SLinus Torvalds found_ok_skb: 16441da177e4SLinus Torvalds /* Ok so how much can we use? */ 16451da177e4SLinus Torvalds used = skb->len - offset; 16461da177e4SLinus Torvalds if (len < used) 16471da177e4SLinus Torvalds used = len; 16481da177e4SLinus Torvalds 16491da177e4SLinus Torvalds /* Do we have urgent data here? */ 16501da177e4SLinus Torvalds if (tp->urg_data) { 16511da177e4SLinus Torvalds u32 urg_offset = tp->urg_seq - *seq; 16521da177e4SLinus Torvalds if (urg_offset < used) { 16531da177e4SLinus Torvalds if (!urg_offset) { 16541da177e4SLinus Torvalds if (!sock_flag(sk, SOCK_URGINLINE)) { 16551da177e4SLinus Torvalds ++*seq; 165677527313SIlpo Järvinen urg_hole++; 16571da177e4SLinus Torvalds offset++; 16581da177e4SLinus Torvalds used--; 16591da177e4SLinus Torvalds if (!used) 16601da177e4SLinus Torvalds goto skip_copy; 16611da177e4SLinus Torvalds } 16621da177e4SLinus Torvalds } else 16631da177e4SLinus Torvalds used = urg_offset; 16641da177e4SLinus Torvalds } 16651da177e4SLinus Torvalds } 16661da177e4SLinus Torvalds 16671da177e4SLinus Torvalds if (!(flags & MSG_TRUNC)) { 16681a2449a8SChris Leech #ifdef CONFIG_NET_DMA 16691a2449a8SChris Leech if (!tp->ucopy.dma_chan && tp->ucopy.pinned_list) 1670a2bd1140SDave Jiang tp->ucopy.dma_chan = net_dma_find_channel(); 16711a2449a8SChris Leech 16721a2449a8SChris Leech if (tp->ucopy.dma_chan) { 16731a2449a8SChris Leech tp->ucopy.dma_cookie = dma_skb_copy_datagram_iovec( 16741a2449a8SChris Leech tp->ucopy.dma_chan, skb, offset, 16751a2449a8SChris Leech msg->msg_iov, used, 16761a2449a8SChris Leech tp->ucopy.pinned_list); 16771a2449a8SChris Leech 16781a2449a8SChris Leech if (tp->ucopy.dma_cookie < 0) { 16791a2449a8SChris Leech 1680afd46503SJoe Perches pr_alert("%s: dma_cookie < 0\n", 1681afd46503SJoe Perches __func__); 16821a2449a8SChris Leech 16831a2449a8SChris Leech /* Exception. Bailout! */ 16841a2449a8SChris Leech if (!copied) 16851a2449a8SChris Leech copied = -EFAULT; 16861a2449a8SChris Leech break; 16871a2449a8SChris Leech } 168873852e81SSteven J. Magnani 168973852e81SSteven J. Magnani dma_async_memcpy_issue_pending(tp->ucopy.dma_chan); 169073852e81SSteven J. Magnani 16911a2449a8SChris Leech if ((offset + used) == skb->len) 16921a2449a8SChris Leech copied_early = 1; 16931a2449a8SChris Leech 16941a2449a8SChris Leech } else 16951a2449a8SChris Leech #endif 16961a2449a8SChris Leech { 16971da177e4SLinus Torvalds err = skb_copy_datagram_iovec(skb, offset, 16981da177e4SLinus Torvalds msg->msg_iov, used); 16991da177e4SLinus Torvalds if (err) { 17001da177e4SLinus Torvalds /* Exception. Bailout! */ 17011da177e4SLinus Torvalds if (!copied) 17021da177e4SLinus Torvalds copied = -EFAULT; 17031da177e4SLinus Torvalds break; 17041da177e4SLinus Torvalds } 17051da177e4SLinus Torvalds } 17061a2449a8SChris Leech } 17071da177e4SLinus Torvalds 17081da177e4SLinus Torvalds *seq += used; 17091da177e4SLinus Torvalds copied += used; 17101da177e4SLinus Torvalds len -= used; 17111da177e4SLinus Torvalds 17121da177e4SLinus Torvalds tcp_rcv_space_adjust(sk); 17131da177e4SLinus Torvalds 17141da177e4SLinus Torvalds skip_copy: 17151da177e4SLinus Torvalds if (tp->urg_data && after(tp->copied_seq, tp->urg_seq)) { 17161da177e4SLinus Torvalds tp->urg_data = 0; 17179e412ba7SIlpo Järvinen tcp_fast_path_check(sk); 17181da177e4SLinus Torvalds } 17191da177e4SLinus Torvalds if (used + offset < skb->len) 17201da177e4SLinus Torvalds continue; 17211da177e4SLinus Torvalds 1722aa8223c7SArnaldo Carvalho de Melo if (tcp_hdr(skb)->fin) 17231da177e4SLinus Torvalds goto found_fin_ok; 17241a2449a8SChris Leech if (!(flags & MSG_PEEK)) { 17251a2449a8SChris Leech sk_eat_skb(sk, skb, copied_early); 17261a2449a8SChris Leech copied_early = 0; 17271a2449a8SChris Leech } 17281da177e4SLinus Torvalds continue; 17291da177e4SLinus Torvalds 17301da177e4SLinus Torvalds found_fin_ok: 17311da177e4SLinus Torvalds /* Process the FIN. */ 17321da177e4SLinus Torvalds ++*seq; 17331a2449a8SChris Leech if (!(flags & MSG_PEEK)) { 17341a2449a8SChris Leech sk_eat_skb(sk, skb, copied_early); 17351a2449a8SChris Leech copied_early = 0; 17361a2449a8SChris Leech } 17371da177e4SLinus Torvalds break; 17381da177e4SLinus Torvalds } while (len > 0); 17391da177e4SLinus Torvalds 17401da177e4SLinus Torvalds if (user_recv) { 1741b03efcfbSDavid S. Miller if (!skb_queue_empty(&tp->ucopy.prequeue)) { 17421da177e4SLinus Torvalds int chunk; 17431da177e4SLinus Torvalds 17441da177e4SLinus Torvalds tp->ucopy.len = copied > 0 ? len : 0; 17451da177e4SLinus Torvalds 17461da177e4SLinus Torvalds tcp_prequeue_process(sk); 17471da177e4SLinus Torvalds 17481da177e4SLinus Torvalds if (copied > 0 && (chunk = len - tp->ucopy.len) != 0) { 1749ed88098eSPavel Emelyanov NET_ADD_STATS_USER(sock_net(sk), LINUX_MIB_TCPDIRECTCOPYFROMPREQUEUE, chunk); 17501da177e4SLinus Torvalds len -= chunk; 17511da177e4SLinus Torvalds copied += chunk; 17521da177e4SLinus Torvalds } 17531da177e4SLinus Torvalds } 17541da177e4SLinus Torvalds 17551da177e4SLinus Torvalds tp->ucopy.task = NULL; 17561da177e4SLinus Torvalds tp->ucopy.len = 0; 17571da177e4SLinus Torvalds } 17581da177e4SLinus Torvalds 17591a2449a8SChris Leech #ifdef CONFIG_NET_DMA 176073852e81SSteven J. Magnani tcp_service_net_dma(sk, true); /* Wait for queue to drain */ 17611a2449a8SChris Leech tp->ucopy.dma_chan = NULL; 176273852e81SSteven J. Magnani 17631a2449a8SChris Leech if (tp->ucopy.pinned_list) { 17641a2449a8SChris Leech dma_unpin_iovec_pages(tp->ucopy.pinned_list); 17651a2449a8SChris Leech tp->ucopy.pinned_list = NULL; 17661a2449a8SChris Leech } 17671a2449a8SChris Leech #endif 17681a2449a8SChris Leech 17691da177e4SLinus Torvalds /* According to UNIX98, msg_name/msg_namelen are ignored 17701da177e4SLinus Torvalds * on connected socket. I was just happy when found this 8) --ANK 17711da177e4SLinus Torvalds */ 17721da177e4SLinus Torvalds 17731da177e4SLinus Torvalds /* Clean up data we have read: This will do ACK frames. */ 17740e4b4992SChris Leech tcp_cleanup_rbuf(sk, copied); 17751da177e4SLinus Torvalds 17761da177e4SLinus Torvalds release_sock(sk); 17771da177e4SLinus Torvalds return copied; 17781da177e4SLinus Torvalds 17791da177e4SLinus Torvalds out: 17801da177e4SLinus Torvalds release_sock(sk); 17811da177e4SLinus Torvalds return err; 17821da177e4SLinus Torvalds 17831da177e4SLinus Torvalds recv_urg: 1784377f0a08SRami Rosen err = tcp_recv_urg(sk, msg, len, flags); 17851da177e4SLinus Torvalds goto out; 17861da177e4SLinus Torvalds } 17874bc2f18bSEric Dumazet EXPORT_SYMBOL(tcp_recvmsg); 17881da177e4SLinus Torvalds 1789490d5046SIlpo Järvinen void tcp_set_state(struct sock *sk, int state) 1790490d5046SIlpo Järvinen { 1791490d5046SIlpo Järvinen int oldstate = sk->sk_state; 1792490d5046SIlpo Järvinen 1793490d5046SIlpo Järvinen switch (state) { 1794490d5046SIlpo Järvinen case TCP_ESTABLISHED: 1795490d5046SIlpo Järvinen if (oldstate != TCP_ESTABLISHED) 179681cc8a75SPavel Emelyanov TCP_INC_STATS(sock_net(sk), TCP_MIB_CURRESTAB); 1797490d5046SIlpo Järvinen break; 1798490d5046SIlpo Järvinen 1799490d5046SIlpo Järvinen case TCP_CLOSE: 1800490d5046SIlpo Järvinen if (oldstate == TCP_CLOSE_WAIT || oldstate == TCP_ESTABLISHED) 180181cc8a75SPavel Emelyanov TCP_INC_STATS(sock_net(sk), TCP_MIB_ESTABRESETS); 1802490d5046SIlpo Järvinen 1803490d5046SIlpo Järvinen sk->sk_prot->unhash(sk); 1804490d5046SIlpo Järvinen if (inet_csk(sk)->icsk_bind_hash && 1805490d5046SIlpo Järvinen !(sk->sk_userlocks & SOCK_BINDPORT_LOCK)) 1806ab1e0a13SArnaldo Carvalho de Melo inet_put_port(sk); 1807490d5046SIlpo Järvinen /* fall through */ 1808490d5046SIlpo Järvinen default: 1809490d5046SIlpo Järvinen if (oldstate == TCP_ESTABLISHED) 181074688e48SPavel Emelyanov TCP_DEC_STATS(sock_net(sk), TCP_MIB_CURRESTAB); 1811490d5046SIlpo Järvinen } 1812490d5046SIlpo Järvinen 1813490d5046SIlpo Järvinen /* Change state AFTER socket is unhashed to avoid closed 1814490d5046SIlpo Järvinen * socket sitting in hash tables. 1815490d5046SIlpo Järvinen */ 1816490d5046SIlpo Järvinen sk->sk_state = state; 1817490d5046SIlpo Järvinen 1818490d5046SIlpo Järvinen #ifdef STATE_TRACE 1819490d5046SIlpo Järvinen SOCK_DEBUG(sk, "TCP sk=%p, State %s -> %s\n", sk, statename[oldstate], statename[state]); 1820490d5046SIlpo Järvinen #endif 1821490d5046SIlpo Järvinen } 1822490d5046SIlpo Järvinen EXPORT_SYMBOL_GPL(tcp_set_state); 1823490d5046SIlpo Järvinen 18241da177e4SLinus Torvalds /* 18251da177e4SLinus Torvalds * State processing on a close. This implements the state shift for 18261da177e4SLinus Torvalds * sending our FIN frame. Note that we only send a FIN for some 18271da177e4SLinus Torvalds * states. A shutdown() may have already sent the FIN, or we may be 18281da177e4SLinus Torvalds * closed. 18291da177e4SLinus Torvalds */ 18301da177e4SLinus Torvalds 18319b5b5cffSArjan van de Ven static const unsigned char new_state[16] = { 18321da177e4SLinus Torvalds /* current state: new state: action: */ 18331da177e4SLinus Torvalds /* (Invalid) */ TCP_CLOSE, 18341da177e4SLinus Torvalds /* TCP_ESTABLISHED */ TCP_FIN_WAIT1 | TCP_ACTION_FIN, 18351da177e4SLinus Torvalds /* TCP_SYN_SENT */ TCP_CLOSE, 18361da177e4SLinus Torvalds /* TCP_SYN_RECV */ TCP_FIN_WAIT1 | TCP_ACTION_FIN, 18371da177e4SLinus Torvalds /* TCP_FIN_WAIT1 */ TCP_FIN_WAIT1, 18381da177e4SLinus Torvalds /* TCP_FIN_WAIT2 */ TCP_FIN_WAIT2, 18391da177e4SLinus Torvalds /* TCP_TIME_WAIT */ TCP_CLOSE, 18401da177e4SLinus Torvalds /* TCP_CLOSE */ TCP_CLOSE, 18411da177e4SLinus Torvalds /* TCP_CLOSE_WAIT */ TCP_LAST_ACK | TCP_ACTION_FIN, 18421da177e4SLinus Torvalds /* TCP_LAST_ACK */ TCP_LAST_ACK, 18431da177e4SLinus Torvalds /* TCP_LISTEN */ TCP_CLOSE, 18441da177e4SLinus Torvalds /* TCP_CLOSING */ TCP_CLOSING, 18451da177e4SLinus Torvalds }; 18461da177e4SLinus Torvalds 18471da177e4SLinus Torvalds static int tcp_close_state(struct sock *sk) 18481da177e4SLinus Torvalds { 18491da177e4SLinus Torvalds int next = (int)new_state[sk->sk_state]; 18501da177e4SLinus Torvalds int ns = next & TCP_STATE_MASK; 18511da177e4SLinus Torvalds 18521da177e4SLinus Torvalds tcp_set_state(sk, ns); 18531da177e4SLinus Torvalds 18541da177e4SLinus Torvalds return next & TCP_ACTION_FIN; 18551da177e4SLinus Torvalds } 18561da177e4SLinus Torvalds 18571da177e4SLinus Torvalds /* 18581da177e4SLinus Torvalds * Shutdown the sending side of a connection. Much like close except 18591f29b058SSatoru SATOH * that we don't receive shut down or sock_set_flag(sk, SOCK_DEAD). 18601da177e4SLinus Torvalds */ 18611da177e4SLinus Torvalds 18621da177e4SLinus Torvalds void tcp_shutdown(struct sock *sk, int how) 18631da177e4SLinus Torvalds { 18641da177e4SLinus Torvalds /* We need to grab some memory, and put together a FIN, 18651da177e4SLinus Torvalds * and then put it into the queue to be sent. 18661da177e4SLinus Torvalds * Tim MacKenzie(tym@dibbler.cs.monash.edu.au) 4 Dec '92. 18671da177e4SLinus Torvalds */ 18681da177e4SLinus Torvalds if (!(how & SEND_SHUTDOWN)) 18691da177e4SLinus Torvalds return; 18701da177e4SLinus Torvalds 18711da177e4SLinus Torvalds /* If we've already sent a FIN, or it's a closed state, skip this. */ 18721da177e4SLinus Torvalds if ((1 << sk->sk_state) & 18731da177e4SLinus Torvalds (TCPF_ESTABLISHED | TCPF_SYN_SENT | 18741da177e4SLinus Torvalds TCPF_SYN_RECV | TCPF_CLOSE_WAIT)) { 18751da177e4SLinus Torvalds /* Clear out any half completed packets. FIN if needed. */ 18761da177e4SLinus Torvalds if (tcp_close_state(sk)) 18771da177e4SLinus Torvalds tcp_send_fin(sk); 18781da177e4SLinus Torvalds } 18791da177e4SLinus Torvalds } 18804bc2f18bSEric Dumazet EXPORT_SYMBOL(tcp_shutdown); 18811da177e4SLinus Torvalds 1882efcdbf24SArun Sharma bool tcp_check_oom(struct sock *sk, int shift) 1883efcdbf24SArun Sharma { 1884efcdbf24SArun Sharma bool too_many_orphans, out_of_socket_memory; 1885efcdbf24SArun Sharma 1886efcdbf24SArun Sharma too_many_orphans = tcp_too_many_orphans(sk, shift); 1887efcdbf24SArun Sharma out_of_socket_memory = tcp_out_of_memory(sk); 1888efcdbf24SArun Sharma 1889efcdbf24SArun Sharma if (too_many_orphans && net_ratelimit()) 1890afd46503SJoe Perches pr_info("too many orphaned sockets\n"); 1891efcdbf24SArun Sharma if (out_of_socket_memory && net_ratelimit()) 1892afd46503SJoe Perches pr_info("out of memory -- consider tuning tcp_mem\n"); 1893efcdbf24SArun Sharma return too_many_orphans || out_of_socket_memory; 1894efcdbf24SArun Sharma } 1895efcdbf24SArun Sharma 18961da177e4SLinus Torvalds void tcp_close(struct sock *sk, long timeout) 18971da177e4SLinus Torvalds { 18981da177e4SLinus Torvalds struct sk_buff *skb; 18991da177e4SLinus Torvalds int data_was_unread = 0; 190075c2d907SHerbert Xu int state; 19011da177e4SLinus Torvalds 19021da177e4SLinus Torvalds lock_sock(sk); 19031da177e4SLinus Torvalds sk->sk_shutdown = SHUTDOWN_MASK; 19041da177e4SLinus Torvalds 19051da177e4SLinus Torvalds if (sk->sk_state == TCP_LISTEN) { 19061da177e4SLinus Torvalds tcp_set_state(sk, TCP_CLOSE); 19071da177e4SLinus Torvalds 19081da177e4SLinus Torvalds /* Special case. */ 19090a5578cfSArnaldo Carvalho de Melo inet_csk_listen_stop(sk); 19101da177e4SLinus Torvalds 19111da177e4SLinus Torvalds goto adjudge_to_death; 19121da177e4SLinus Torvalds } 19131da177e4SLinus Torvalds 19141da177e4SLinus Torvalds /* We need to flush the recv. buffs. We do this only on the 19151da177e4SLinus Torvalds * descriptor close, not protocol-sourced closes, because the 19161da177e4SLinus Torvalds * reader process may not have drained the data yet! 19171da177e4SLinus Torvalds */ 19181da177e4SLinus Torvalds while ((skb = __skb_dequeue(&sk->sk_receive_queue)) != NULL) { 19191da177e4SLinus Torvalds u32 len = TCP_SKB_CB(skb)->end_seq - TCP_SKB_CB(skb)->seq - 1920aa8223c7SArnaldo Carvalho de Melo tcp_hdr(skb)->fin; 19211da177e4SLinus Torvalds data_was_unread += len; 19221da177e4SLinus Torvalds __kfree_skb(skb); 19231da177e4SLinus Torvalds } 19241da177e4SLinus Torvalds 19253ab224beSHideo Aoki sk_mem_reclaim(sk); 19261da177e4SLinus Torvalds 1927565b7b2dSKonstantin Khorenko /* If socket has been already reset (e.g. in tcp_reset()) - kill it. */ 1928565b7b2dSKonstantin Khorenko if (sk->sk_state == TCP_CLOSE) 1929565b7b2dSKonstantin Khorenko goto adjudge_to_death; 1930565b7b2dSKonstantin Khorenko 193165bb723cSGerrit Renker /* As outlined in RFC 2525, section 2.17, we send a RST here because 193265bb723cSGerrit Renker * data was lost. To witness the awful effects of the old behavior of 193365bb723cSGerrit Renker * always doing a FIN, run an older 2.1.x kernel or 2.0.x, start a bulk 193465bb723cSGerrit Renker * GET in an FTP client, suspend the process, wait for the client to 193565bb723cSGerrit Renker * advertise a zero window, then kill -9 the FTP client, wheee... 193665bb723cSGerrit Renker * Note: timeout is always zero in such a case. 19371da177e4SLinus Torvalds */ 1938*ee995283SPavel Emelyanov if (unlikely(tcp_sk(sk)->repair)) { 1939*ee995283SPavel Emelyanov sk->sk_prot->disconnect(sk, 0); 1940*ee995283SPavel Emelyanov } else if (data_was_unread) { 19411da177e4SLinus Torvalds /* Unread data was tossed, zap the connection. */ 19426f67c817SPavel Emelyanov NET_INC_STATS_USER(sock_net(sk), LINUX_MIB_TCPABORTONCLOSE); 19431da177e4SLinus Torvalds tcp_set_state(sk, TCP_CLOSE); 1944aa133076SWu Fengguang tcp_send_active_reset(sk, sk->sk_allocation); 19451da177e4SLinus Torvalds } else if (sock_flag(sk, SOCK_LINGER) && !sk->sk_lingertime) { 19461da177e4SLinus Torvalds /* Check zero linger _after_ checking for unread data. */ 19471da177e4SLinus Torvalds sk->sk_prot->disconnect(sk, 0); 19486f67c817SPavel Emelyanov NET_INC_STATS_USER(sock_net(sk), LINUX_MIB_TCPABORTONDATA); 19491da177e4SLinus Torvalds } else if (tcp_close_state(sk)) { 19501da177e4SLinus Torvalds /* We FIN if the application ate all the data before 19511da177e4SLinus Torvalds * zapping the connection. 19521da177e4SLinus Torvalds */ 19531da177e4SLinus Torvalds 19541da177e4SLinus Torvalds /* RED-PEN. Formally speaking, we have broken TCP state 19551da177e4SLinus Torvalds * machine. State transitions: 19561da177e4SLinus Torvalds * 19571da177e4SLinus Torvalds * TCP_ESTABLISHED -> TCP_FIN_WAIT1 19581da177e4SLinus Torvalds * TCP_SYN_RECV -> TCP_FIN_WAIT1 (forget it, it's impossible) 19591da177e4SLinus Torvalds * TCP_CLOSE_WAIT -> TCP_LAST_ACK 19601da177e4SLinus Torvalds * 19611da177e4SLinus Torvalds * are legal only when FIN has been sent (i.e. in window), 19621da177e4SLinus Torvalds * rather than queued out of window. Purists blame. 19631da177e4SLinus Torvalds * 19641da177e4SLinus Torvalds * F.e. "RFC state" is ESTABLISHED, 19651da177e4SLinus Torvalds * if Linux state is FIN-WAIT-1, but FIN is still not sent. 19661da177e4SLinus Torvalds * 19671da177e4SLinus Torvalds * The visible declinations are that sometimes 19681da177e4SLinus Torvalds * we enter time-wait state, when it is not required really 19691da177e4SLinus Torvalds * (harmless), do not send active resets, when they are 19701da177e4SLinus Torvalds * required by specs (TCP_ESTABLISHED, TCP_CLOSE_WAIT, when 19711da177e4SLinus Torvalds * they look as CLOSING or LAST_ACK for Linux) 19721da177e4SLinus Torvalds * Probably, I missed some more holelets. 19731da177e4SLinus Torvalds * --ANK 19741da177e4SLinus Torvalds */ 19751da177e4SLinus Torvalds tcp_send_fin(sk); 19761da177e4SLinus Torvalds } 19771da177e4SLinus Torvalds 19781da177e4SLinus Torvalds sk_stream_wait_close(sk, timeout); 19791da177e4SLinus Torvalds 19801da177e4SLinus Torvalds adjudge_to_death: 198175c2d907SHerbert Xu state = sk->sk_state; 198275c2d907SHerbert Xu sock_hold(sk); 198375c2d907SHerbert Xu sock_orphan(sk); 198475c2d907SHerbert Xu 19851da177e4SLinus Torvalds /* It is the last release_sock in its life. It will remove backlog. */ 19861da177e4SLinus Torvalds release_sock(sk); 19871da177e4SLinus Torvalds 19881da177e4SLinus Torvalds 19891da177e4SLinus Torvalds /* Now socket is owned by kernel and we acquire BH lock 19901da177e4SLinus Torvalds to finish close. No need to check for user refs. 19911da177e4SLinus Torvalds */ 19921da177e4SLinus Torvalds local_bh_disable(); 19931da177e4SLinus Torvalds bh_lock_sock(sk); 1994547b792cSIlpo Järvinen WARN_ON(sock_owned_by_user(sk)); 19951da177e4SLinus Torvalds 1996eb4dea58SHerbert Xu percpu_counter_inc(sk->sk_prot->orphan_count); 1997eb4dea58SHerbert Xu 199875c2d907SHerbert Xu /* Have we already been destroyed by a softirq or backlog? */ 199975c2d907SHerbert Xu if (state != TCP_CLOSE && sk->sk_state == TCP_CLOSE) 200075c2d907SHerbert Xu goto out; 20011da177e4SLinus Torvalds 20021da177e4SLinus Torvalds /* This is a (useful) BSD violating of the RFC. There is a 20031da177e4SLinus Torvalds * problem with TCP as specified in that the other end could 20041da177e4SLinus Torvalds * keep a socket open forever with no application left this end. 20051da177e4SLinus Torvalds * We use a 3 minute timeout (about the same as BSD) then kill 20061da177e4SLinus Torvalds * our end. If they send after that then tough - BUT: long enough 20071da177e4SLinus Torvalds * that we won't make the old 4*rto = almost no time - whoops 20081da177e4SLinus Torvalds * reset mistake. 20091da177e4SLinus Torvalds * 20101da177e4SLinus Torvalds * Nope, it was not mistake. It is really desired behaviour 20111da177e4SLinus Torvalds * f.e. on http servers, when such sockets are useless, but 20121da177e4SLinus Torvalds * consume significant resources. Let's do it with special 20131da177e4SLinus Torvalds * linger2 option. --ANK 20141da177e4SLinus Torvalds */ 20151da177e4SLinus Torvalds 20161da177e4SLinus Torvalds if (sk->sk_state == TCP_FIN_WAIT2) { 20171da177e4SLinus Torvalds struct tcp_sock *tp = tcp_sk(sk); 20181da177e4SLinus Torvalds if (tp->linger2 < 0) { 20191da177e4SLinus Torvalds tcp_set_state(sk, TCP_CLOSE); 20201da177e4SLinus Torvalds tcp_send_active_reset(sk, GFP_ATOMIC); 2021de0744afSPavel Emelyanov NET_INC_STATS_BH(sock_net(sk), 2022de0744afSPavel Emelyanov LINUX_MIB_TCPABORTONLINGER); 20231da177e4SLinus Torvalds } else { 2024463c84b9SArnaldo Carvalho de Melo const int tmo = tcp_fin_time(sk); 20251da177e4SLinus Torvalds 20261da177e4SLinus Torvalds if (tmo > TCP_TIMEWAIT_LEN) { 202752499afeSDavid S. Miller inet_csk_reset_keepalive_timer(sk, 202852499afeSDavid S. Miller tmo - TCP_TIMEWAIT_LEN); 20291da177e4SLinus Torvalds } else { 20301da177e4SLinus Torvalds tcp_time_wait(sk, TCP_FIN_WAIT2, tmo); 20311da177e4SLinus Torvalds goto out; 20321da177e4SLinus Torvalds } 20331da177e4SLinus Torvalds } 20341da177e4SLinus Torvalds } 20351da177e4SLinus Torvalds if (sk->sk_state != TCP_CLOSE) { 20363ab224beSHideo Aoki sk_mem_reclaim(sk); 2037efcdbf24SArun Sharma if (tcp_check_oom(sk, 0)) { 20381da177e4SLinus Torvalds tcp_set_state(sk, TCP_CLOSE); 20391da177e4SLinus Torvalds tcp_send_active_reset(sk, GFP_ATOMIC); 2040de0744afSPavel Emelyanov NET_INC_STATS_BH(sock_net(sk), 2041de0744afSPavel Emelyanov LINUX_MIB_TCPABORTONMEMORY); 20421da177e4SLinus Torvalds } 20431da177e4SLinus Torvalds } 20441da177e4SLinus Torvalds 20451da177e4SLinus Torvalds if (sk->sk_state == TCP_CLOSE) 20460a5578cfSArnaldo Carvalho de Melo inet_csk_destroy_sock(sk); 20471da177e4SLinus Torvalds /* Otherwise, socket is reprieved until protocol close. */ 20481da177e4SLinus Torvalds 20491da177e4SLinus Torvalds out: 20501da177e4SLinus Torvalds bh_unlock_sock(sk); 20511da177e4SLinus Torvalds local_bh_enable(); 20521da177e4SLinus Torvalds sock_put(sk); 20531da177e4SLinus Torvalds } 20544bc2f18bSEric Dumazet EXPORT_SYMBOL(tcp_close); 20551da177e4SLinus Torvalds 20561da177e4SLinus Torvalds /* These states need RST on ABORT according to RFC793 */ 20571da177e4SLinus Torvalds 20581da177e4SLinus Torvalds static inline int tcp_need_reset(int state) 20591da177e4SLinus Torvalds { 20601da177e4SLinus Torvalds return (1 << state) & 20611da177e4SLinus Torvalds (TCPF_ESTABLISHED | TCPF_CLOSE_WAIT | TCPF_FIN_WAIT1 | 20621da177e4SLinus Torvalds TCPF_FIN_WAIT2 | TCPF_SYN_RECV); 20631da177e4SLinus Torvalds } 20641da177e4SLinus Torvalds 20651da177e4SLinus Torvalds int tcp_disconnect(struct sock *sk, int flags) 20661da177e4SLinus Torvalds { 20671da177e4SLinus Torvalds struct inet_sock *inet = inet_sk(sk); 2068463c84b9SArnaldo Carvalho de Melo struct inet_connection_sock *icsk = inet_csk(sk); 20691da177e4SLinus Torvalds struct tcp_sock *tp = tcp_sk(sk); 20701da177e4SLinus Torvalds int err = 0; 20711da177e4SLinus Torvalds int old_state = sk->sk_state; 20721da177e4SLinus Torvalds 20731da177e4SLinus Torvalds if (old_state != TCP_CLOSE) 20741da177e4SLinus Torvalds tcp_set_state(sk, TCP_CLOSE); 20751da177e4SLinus Torvalds 20761da177e4SLinus Torvalds /* ABORT function of RFC793 */ 20771da177e4SLinus Torvalds if (old_state == TCP_LISTEN) { 20780a5578cfSArnaldo Carvalho de Melo inet_csk_listen_stop(sk); 2079*ee995283SPavel Emelyanov } else if (unlikely(tp->repair)) { 2080*ee995283SPavel Emelyanov sk->sk_err = ECONNABORTED; 20811da177e4SLinus Torvalds } else if (tcp_need_reset(old_state) || 20821da177e4SLinus Torvalds (tp->snd_nxt != tp->write_seq && 20831da177e4SLinus Torvalds (1 << old_state) & (TCPF_CLOSING | TCPF_LAST_ACK))) { 2084caa20d9aSStephen Hemminger /* The last check adjusts for discrepancy of Linux wrt. RFC 20851da177e4SLinus Torvalds * states 20861da177e4SLinus Torvalds */ 20871da177e4SLinus Torvalds tcp_send_active_reset(sk, gfp_any()); 20881da177e4SLinus Torvalds sk->sk_err = ECONNRESET; 20891da177e4SLinus Torvalds } else if (old_state == TCP_SYN_SENT) 20901da177e4SLinus Torvalds sk->sk_err = ECONNRESET; 20911da177e4SLinus Torvalds 20921da177e4SLinus Torvalds tcp_clear_xmit_timers(sk); 20931da177e4SLinus Torvalds __skb_queue_purge(&sk->sk_receive_queue); 2094fe067e8aSDavid S. Miller tcp_write_queue_purge(sk); 20951da177e4SLinus Torvalds __skb_queue_purge(&tp->out_of_order_queue); 20961a2449a8SChris Leech #ifdef CONFIG_NET_DMA 20971a2449a8SChris Leech __skb_queue_purge(&sk->sk_async_wait_queue); 20981a2449a8SChris Leech #endif 20991da177e4SLinus Torvalds 2100c720c7e8SEric Dumazet inet->inet_dport = 0; 21011da177e4SLinus Torvalds 21021da177e4SLinus Torvalds if (!(sk->sk_userlocks & SOCK_BINDADDR_LOCK)) 21031da177e4SLinus Torvalds inet_reset_saddr(sk); 21041da177e4SLinus Torvalds 21051da177e4SLinus Torvalds sk->sk_shutdown = 0; 21061da177e4SLinus Torvalds sock_reset_flag(sk, SOCK_DONE); 21071da177e4SLinus Torvalds tp->srtt = 0; 21081da177e4SLinus Torvalds if ((tp->write_seq += tp->max_window + 2) == 0) 21091da177e4SLinus Torvalds tp->write_seq = 1; 2110463c84b9SArnaldo Carvalho de Melo icsk->icsk_backoff = 0; 21111da177e4SLinus Torvalds tp->snd_cwnd = 2; 21126687e988SArnaldo Carvalho de Melo icsk->icsk_probes_out = 0; 21131da177e4SLinus Torvalds tp->packets_out = 0; 21140b6a05c1SIlpo Järvinen tp->snd_ssthresh = TCP_INFINITE_SSTHRESH; 21151da177e4SLinus Torvalds tp->snd_cwnd_cnt = 0; 21169772efb9SStephen Hemminger tp->bytes_acked = 0; 21171fdf475aSEric Dumazet tp->window_clamp = 0; 21186687e988SArnaldo Carvalho de Melo tcp_set_ca_state(sk, TCP_CA_Open); 21191da177e4SLinus Torvalds tcp_clear_retrans(tp); 2120463c84b9SArnaldo Carvalho de Melo inet_csk_delack_init(sk); 2121fe067e8aSDavid S. Miller tcp_init_send_head(sk); 2122b40b4f79SSrinivas Aji memset(&tp->rx_opt, 0, sizeof(tp->rx_opt)); 21231da177e4SLinus Torvalds __sk_dst_reset(sk); 21241da177e4SLinus Torvalds 2125c720c7e8SEric Dumazet WARN_ON(inet->inet_num && !icsk->icsk_bind_hash); 21261da177e4SLinus Torvalds 21271da177e4SLinus Torvalds sk->sk_error_report(sk); 21281da177e4SLinus Torvalds return err; 21291da177e4SLinus Torvalds } 21304bc2f18bSEric Dumazet EXPORT_SYMBOL(tcp_disconnect); 21311da177e4SLinus Torvalds 2132*ee995283SPavel Emelyanov static inline int tcp_can_repair_sock(struct sock *sk) 2133*ee995283SPavel Emelyanov { 2134*ee995283SPavel Emelyanov return capable(CAP_NET_ADMIN) && 2135*ee995283SPavel Emelyanov ((1 << sk->sk_state) & (TCPF_CLOSE | TCPF_ESTABLISHED)); 2136*ee995283SPavel Emelyanov } 2137*ee995283SPavel Emelyanov 21381da177e4SLinus Torvalds /* 21391da177e4SLinus Torvalds * Socket option code for TCP. 21401da177e4SLinus Torvalds */ 21413fdadf7dSDmitry Mishin static int do_tcp_setsockopt(struct sock *sk, int level, 2142b7058842SDavid S. Miller int optname, char __user *optval, unsigned int optlen) 21431da177e4SLinus Torvalds { 21441da177e4SLinus Torvalds struct tcp_sock *tp = tcp_sk(sk); 2145463c84b9SArnaldo Carvalho de Melo struct inet_connection_sock *icsk = inet_csk(sk); 21461da177e4SLinus Torvalds int val; 21471da177e4SLinus Torvalds int err = 0; 21481da177e4SLinus Torvalds 2149e56fb50fSWilliam Allen Simpson /* These are data/string values, all the others are ints */ 2150e56fb50fSWilliam Allen Simpson switch (optname) { 2151e56fb50fSWilliam Allen Simpson case TCP_CONGESTION: { 21525f8ef48dSStephen Hemminger char name[TCP_CA_NAME_MAX]; 21535f8ef48dSStephen Hemminger 21545f8ef48dSStephen Hemminger if (optlen < 1) 21555f8ef48dSStephen Hemminger return -EINVAL; 21565f8ef48dSStephen Hemminger 21575f8ef48dSStephen Hemminger val = strncpy_from_user(name, optval, 21584fdb78d3SAndrew Morton min_t(long, TCP_CA_NAME_MAX-1, optlen)); 21595f8ef48dSStephen Hemminger if (val < 0) 21605f8ef48dSStephen Hemminger return -EFAULT; 21615f8ef48dSStephen Hemminger name[val] = 0; 21625f8ef48dSStephen Hemminger 21635f8ef48dSStephen Hemminger lock_sock(sk); 21646687e988SArnaldo Carvalho de Melo err = tcp_set_congestion_control(sk, name); 21655f8ef48dSStephen Hemminger release_sock(sk); 21665f8ef48dSStephen Hemminger return err; 21675f8ef48dSStephen Hemminger } 2168e56fb50fSWilliam Allen Simpson case TCP_COOKIE_TRANSACTIONS: { 2169e56fb50fSWilliam Allen Simpson struct tcp_cookie_transactions ctd; 2170e56fb50fSWilliam Allen Simpson struct tcp_cookie_values *cvp = NULL; 2171e56fb50fSWilliam Allen Simpson 2172e56fb50fSWilliam Allen Simpson if (sizeof(ctd) > optlen) 2173e56fb50fSWilliam Allen Simpson return -EINVAL; 2174e56fb50fSWilliam Allen Simpson if (copy_from_user(&ctd, optval, sizeof(ctd))) 2175e56fb50fSWilliam Allen Simpson return -EFAULT; 2176e56fb50fSWilliam Allen Simpson 2177e56fb50fSWilliam Allen Simpson if (ctd.tcpct_used > sizeof(ctd.tcpct_value) || 2178e56fb50fSWilliam Allen Simpson ctd.tcpct_s_data_desired > TCP_MSS_DESIRED) 2179e56fb50fSWilliam Allen Simpson return -EINVAL; 2180e56fb50fSWilliam Allen Simpson 2181e56fb50fSWilliam Allen Simpson if (ctd.tcpct_cookie_desired == 0) { 2182e56fb50fSWilliam Allen Simpson /* default to global value */ 2183e56fb50fSWilliam Allen Simpson } else if ((0x1 & ctd.tcpct_cookie_desired) || 2184e56fb50fSWilliam Allen Simpson ctd.tcpct_cookie_desired > TCP_COOKIE_MAX || 2185e56fb50fSWilliam Allen Simpson ctd.tcpct_cookie_desired < TCP_COOKIE_MIN) { 2186e56fb50fSWilliam Allen Simpson return -EINVAL; 2187e56fb50fSWilliam Allen Simpson } 2188e56fb50fSWilliam Allen Simpson 2189e56fb50fSWilliam Allen Simpson if (TCP_COOKIE_OUT_NEVER & ctd.tcpct_flags) { 2190e56fb50fSWilliam Allen Simpson /* Supercedes all other values */ 2191e56fb50fSWilliam Allen Simpson lock_sock(sk); 2192e56fb50fSWilliam Allen Simpson if (tp->cookie_values != NULL) { 2193e56fb50fSWilliam Allen Simpson kref_put(&tp->cookie_values->kref, 2194e56fb50fSWilliam Allen Simpson tcp_cookie_values_release); 2195e56fb50fSWilliam Allen Simpson tp->cookie_values = NULL; 2196e56fb50fSWilliam Allen Simpson } 2197e56fb50fSWilliam Allen Simpson tp->rx_opt.cookie_in_always = 0; /* false */ 2198e56fb50fSWilliam Allen Simpson tp->rx_opt.cookie_out_never = 1; /* true */ 2199e56fb50fSWilliam Allen Simpson release_sock(sk); 2200e56fb50fSWilliam Allen Simpson return err; 2201e56fb50fSWilliam Allen Simpson } 2202e56fb50fSWilliam Allen Simpson 2203e56fb50fSWilliam Allen Simpson /* Allocate ancillary memory before locking. 2204e56fb50fSWilliam Allen Simpson */ 2205e56fb50fSWilliam Allen Simpson if (ctd.tcpct_used > 0 || 2206e56fb50fSWilliam Allen Simpson (tp->cookie_values == NULL && 2207e56fb50fSWilliam Allen Simpson (sysctl_tcp_cookie_size > 0 || 2208e56fb50fSWilliam Allen Simpson ctd.tcpct_cookie_desired > 0 || 2209e56fb50fSWilliam Allen Simpson ctd.tcpct_s_data_desired > 0))) { 2210e56fb50fSWilliam Allen Simpson cvp = kzalloc(sizeof(*cvp) + ctd.tcpct_used, 2211e56fb50fSWilliam Allen Simpson GFP_KERNEL); 2212e56fb50fSWilliam Allen Simpson if (cvp == NULL) 2213e56fb50fSWilliam Allen Simpson return -ENOMEM; 2214a3bdb549SDmitry Popov 2215a3bdb549SDmitry Popov kref_init(&cvp->kref); 2216e56fb50fSWilliam Allen Simpson } 2217e56fb50fSWilliam Allen Simpson lock_sock(sk); 2218e56fb50fSWilliam Allen Simpson tp->rx_opt.cookie_in_always = 2219e56fb50fSWilliam Allen Simpson (TCP_COOKIE_IN_ALWAYS & ctd.tcpct_flags); 2220e56fb50fSWilliam Allen Simpson tp->rx_opt.cookie_out_never = 0; /* false */ 2221e56fb50fSWilliam Allen Simpson 2222e56fb50fSWilliam Allen Simpson if (tp->cookie_values != NULL) { 2223e56fb50fSWilliam Allen Simpson if (cvp != NULL) { 2224e56fb50fSWilliam Allen Simpson /* Changed values are recorded by a changed 2225e56fb50fSWilliam Allen Simpson * pointer, ensuring the cookie will differ, 2226e56fb50fSWilliam Allen Simpson * without separately hashing each value later. 2227e56fb50fSWilliam Allen Simpson */ 2228e56fb50fSWilliam Allen Simpson kref_put(&tp->cookie_values->kref, 2229e56fb50fSWilliam Allen Simpson tcp_cookie_values_release); 2230e56fb50fSWilliam Allen Simpson } else { 2231e56fb50fSWilliam Allen Simpson cvp = tp->cookie_values; 2232e56fb50fSWilliam Allen Simpson } 2233e56fb50fSWilliam Allen Simpson } 2234a3bdb549SDmitry Popov 2235e56fb50fSWilliam Allen Simpson if (cvp != NULL) { 2236e56fb50fSWilliam Allen Simpson cvp->cookie_desired = ctd.tcpct_cookie_desired; 2237e56fb50fSWilliam Allen Simpson 2238e56fb50fSWilliam Allen Simpson if (ctd.tcpct_used > 0) { 2239e56fb50fSWilliam Allen Simpson memcpy(cvp->s_data_payload, ctd.tcpct_value, 2240e56fb50fSWilliam Allen Simpson ctd.tcpct_used); 2241e56fb50fSWilliam Allen Simpson cvp->s_data_desired = ctd.tcpct_used; 2242e56fb50fSWilliam Allen Simpson cvp->s_data_constant = 1; /* true */ 2243e56fb50fSWilliam Allen Simpson } else { 2244e56fb50fSWilliam Allen Simpson /* No constant payload data. */ 2245e56fb50fSWilliam Allen Simpson cvp->s_data_desired = ctd.tcpct_s_data_desired; 2246e56fb50fSWilliam Allen Simpson cvp->s_data_constant = 0; /* false */ 2247e56fb50fSWilliam Allen Simpson } 2248a3bdb549SDmitry Popov 2249a3bdb549SDmitry Popov tp->cookie_values = cvp; 2250e56fb50fSWilliam Allen Simpson } 2251e56fb50fSWilliam Allen Simpson release_sock(sk); 2252e56fb50fSWilliam Allen Simpson return err; 2253e56fb50fSWilliam Allen Simpson } 2254e56fb50fSWilliam Allen Simpson default: 2255e56fb50fSWilliam Allen Simpson /* fallthru */ 2256e56fb50fSWilliam Allen Simpson break; 2257ccbd6a5aSJoe Perches } 22585f8ef48dSStephen Hemminger 22591da177e4SLinus Torvalds if (optlen < sizeof(int)) 22601da177e4SLinus Torvalds return -EINVAL; 22611da177e4SLinus Torvalds 22621da177e4SLinus Torvalds if (get_user(val, (int __user *)optval)) 22631da177e4SLinus Torvalds return -EFAULT; 22641da177e4SLinus Torvalds 22651da177e4SLinus Torvalds lock_sock(sk); 22661da177e4SLinus Torvalds 22671da177e4SLinus Torvalds switch (optname) { 22681da177e4SLinus Torvalds case TCP_MAXSEG: 22691da177e4SLinus Torvalds /* Values greater than interface MTU won't take effect. However 22701da177e4SLinus Torvalds * at the point when this call is done we typically don't yet 22711da177e4SLinus Torvalds * know which interface is going to be used */ 2272c39508d6SDavid S. Miller if (val < TCP_MIN_MSS || val > MAX_TCP_WINDOW) { 22731da177e4SLinus Torvalds err = -EINVAL; 22741da177e4SLinus Torvalds break; 22751da177e4SLinus Torvalds } 22761da177e4SLinus Torvalds tp->rx_opt.user_mss = val; 22771da177e4SLinus Torvalds break; 22781da177e4SLinus Torvalds 22791da177e4SLinus Torvalds case TCP_NODELAY: 22801da177e4SLinus Torvalds if (val) { 22811da177e4SLinus Torvalds /* TCP_NODELAY is weaker than TCP_CORK, so that 22821da177e4SLinus Torvalds * this option on corked socket is remembered, but 22831da177e4SLinus Torvalds * it is not activated until cork is cleared. 22841da177e4SLinus Torvalds * 22851da177e4SLinus Torvalds * However, when TCP_NODELAY is set we make 22861da177e4SLinus Torvalds * an explicit push, which overrides even TCP_CORK 22871da177e4SLinus Torvalds * for currently queued segments. 22881da177e4SLinus Torvalds */ 22891da177e4SLinus Torvalds tp->nonagle |= TCP_NAGLE_OFF|TCP_NAGLE_PUSH; 22909e412ba7SIlpo Järvinen tcp_push_pending_frames(sk); 22911da177e4SLinus Torvalds } else { 22921da177e4SLinus Torvalds tp->nonagle &= ~TCP_NAGLE_OFF; 22931da177e4SLinus Torvalds } 22941da177e4SLinus Torvalds break; 22951da177e4SLinus Torvalds 229636e31b0aSAndreas Petlund case TCP_THIN_LINEAR_TIMEOUTS: 229736e31b0aSAndreas Petlund if (val < 0 || val > 1) 229836e31b0aSAndreas Petlund err = -EINVAL; 229936e31b0aSAndreas Petlund else 230036e31b0aSAndreas Petlund tp->thin_lto = val; 230136e31b0aSAndreas Petlund break; 230236e31b0aSAndreas Petlund 23037e380175SAndreas Petlund case TCP_THIN_DUPACK: 23047e380175SAndreas Petlund if (val < 0 || val > 1) 23057e380175SAndreas Petlund err = -EINVAL; 23067e380175SAndreas Petlund else 23077e380175SAndreas Petlund tp->thin_dupack = val; 23087e380175SAndreas Petlund break; 23097e380175SAndreas Petlund 2310*ee995283SPavel Emelyanov case TCP_REPAIR: 2311*ee995283SPavel Emelyanov if (!tcp_can_repair_sock(sk)) 2312*ee995283SPavel Emelyanov err = -EPERM; 2313*ee995283SPavel Emelyanov else if (val == 1) { 2314*ee995283SPavel Emelyanov tp->repair = 1; 2315*ee995283SPavel Emelyanov sk->sk_reuse = SK_FORCE_REUSE; 2316*ee995283SPavel Emelyanov tp->repair_queue = TCP_NO_QUEUE; 2317*ee995283SPavel Emelyanov } else if (val == 0) { 2318*ee995283SPavel Emelyanov tp->repair = 0; 2319*ee995283SPavel Emelyanov sk->sk_reuse = SK_NO_REUSE; 2320*ee995283SPavel Emelyanov tcp_send_window_probe(sk); 2321*ee995283SPavel Emelyanov } else 2322*ee995283SPavel Emelyanov err = -EINVAL; 2323*ee995283SPavel Emelyanov 2324*ee995283SPavel Emelyanov break; 2325*ee995283SPavel Emelyanov 2326*ee995283SPavel Emelyanov case TCP_REPAIR_QUEUE: 2327*ee995283SPavel Emelyanov if (!tp->repair) 2328*ee995283SPavel Emelyanov err = -EPERM; 2329*ee995283SPavel Emelyanov else if (val < TCP_QUEUES_NR) 2330*ee995283SPavel Emelyanov tp->repair_queue = val; 2331*ee995283SPavel Emelyanov else 2332*ee995283SPavel Emelyanov err = -EINVAL; 2333*ee995283SPavel Emelyanov break; 2334*ee995283SPavel Emelyanov 2335*ee995283SPavel Emelyanov case TCP_QUEUE_SEQ: 2336*ee995283SPavel Emelyanov if (sk->sk_state != TCP_CLOSE) 2337*ee995283SPavel Emelyanov err = -EPERM; 2338*ee995283SPavel Emelyanov else if (tp->repair_queue == TCP_SEND_QUEUE) 2339*ee995283SPavel Emelyanov tp->write_seq = val; 2340*ee995283SPavel Emelyanov else if (tp->repair_queue == TCP_RECV_QUEUE) 2341*ee995283SPavel Emelyanov tp->rcv_nxt = val; 2342*ee995283SPavel Emelyanov else 2343*ee995283SPavel Emelyanov err = -EINVAL; 2344*ee995283SPavel Emelyanov break; 2345*ee995283SPavel Emelyanov 23461da177e4SLinus Torvalds case TCP_CORK: 23471da177e4SLinus Torvalds /* When set indicates to always queue non-full frames. 23481da177e4SLinus Torvalds * Later the user clears this option and we transmit 23491da177e4SLinus Torvalds * any pending partial frames in the queue. This is 23501da177e4SLinus Torvalds * meant to be used alongside sendfile() to get properly 23511da177e4SLinus Torvalds * filled frames when the user (for example) must write 23521da177e4SLinus Torvalds * out headers with a write() call first and then use 23531da177e4SLinus Torvalds * sendfile to send out the data parts. 23541da177e4SLinus Torvalds * 23551da177e4SLinus Torvalds * TCP_CORK can be set together with TCP_NODELAY and it is 23561da177e4SLinus Torvalds * stronger than TCP_NODELAY. 23571da177e4SLinus Torvalds */ 23581da177e4SLinus Torvalds if (val) { 23591da177e4SLinus Torvalds tp->nonagle |= TCP_NAGLE_CORK; 23601da177e4SLinus Torvalds } else { 23611da177e4SLinus Torvalds tp->nonagle &= ~TCP_NAGLE_CORK; 23621da177e4SLinus Torvalds if (tp->nonagle&TCP_NAGLE_OFF) 23631da177e4SLinus Torvalds tp->nonagle |= TCP_NAGLE_PUSH; 23649e412ba7SIlpo Järvinen tcp_push_pending_frames(sk); 23651da177e4SLinus Torvalds } 23661da177e4SLinus Torvalds break; 23671da177e4SLinus Torvalds 23681da177e4SLinus Torvalds case TCP_KEEPIDLE: 23691da177e4SLinus Torvalds if (val < 1 || val > MAX_TCP_KEEPIDLE) 23701da177e4SLinus Torvalds err = -EINVAL; 23711da177e4SLinus Torvalds else { 23721da177e4SLinus Torvalds tp->keepalive_time = val * HZ; 23731da177e4SLinus Torvalds if (sock_flag(sk, SOCK_KEEPOPEN) && 23741da177e4SLinus Torvalds !((1 << sk->sk_state) & 23751da177e4SLinus Torvalds (TCPF_CLOSE | TCPF_LISTEN))) { 23766c37e5deSFlavio Leitner u32 elapsed = keepalive_time_elapsed(tp); 23771da177e4SLinus Torvalds if (tp->keepalive_time > elapsed) 23781da177e4SLinus Torvalds elapsed = tp->keepalive_time - elapsed; 23791da177e4SLinus Torvalds else 23801da177e4SLinus Torvalds elapsed = 0; 2381463c84b9SArnaldo Carvalho de Melo inet_csk_reset_keepalive_timer(sk, elapsed); 23821da177e4SLinus Torvalds } 23831da177e4SLinus Torvalds } 23841da177e4SLinus Torvalds break; 23851da177e4SLinus Torvalds case TCP_KEEPINTVL: 23861da177e4SLinus Torvalds if (val < 1 || val > MAX_TCP_KEEPINTVL) 23871da177e4SLinus Torvalds err = -EINVAL; 23881da177e4SLinus Torvalds else 23891da177e4SLinus Torvalds tp->keepalive_intvl = val * HZ; 23901da177e4SLinus Torvalds break; 23911da177e4SLinus Torvalds case TCP_KEEPCNT: 23921da177e4SLinus Torvalds if (val < 1 || val > MAX_TCP_KEEPCNT) 23931da177e4SLinus Torvalds err = -EINVAL; 23941da177e4SLinus Torvalds else 23951da177e4SLinus Torvalds tp->keepalive_probes = val; 23961da177e4SLinus Torvalds break; 23971da177e4SLinus Torvalds case TCP_SYNCNT: 23981da177e4SLinus Torvalds if (val < 1 || val > MAX_TCP_SYNCNT) 23991da177e4SLinus Torvalds err = -EINVAL; 24001da177e4SLinus Torvalds else 2401463c84b9SArnaldo Carvalho de Melo icsk->icsk_syn_retries = val; 24021da177e4SLinus Torvalds break; 24031da177e4SLinus Torvalds 24041da177e4SLinus Torvalds case TCP_LINGER2: 24051da177e4SLinus Torvalds if (val < 0) 24061da177e4SLinus Torvalds tp->linger2 = -1; 24071da177e4SLinus Torvalds else if (val > sysctl_tcp_fin_timeout / HZ) 24081da177e4SLinus Torvalds tp->linger2 = 0; 24091da177e4SLinus Torvalds else 24101da177e4SLinus Torvalds tp->linger2 = val * HZ; 24111da177e4SLinus Torvalds break; 24121da177e4SLinus Torvalds 24131da177e4SLinus Torvalds case TCP_DEFER_ACCEPT: 2414b103cf34SJulian Anastasov /* Translate value in seconds to number of retransmits */ 2415b103cf34SJulian Anastasov icsk->icsk_accept_queue.rskq_defer_accept = 2416b103cf34SJulian Anastasov secs_to_retrans(val, TCP_TIMEOUT_INIT / HZ, 2417b103cf34SJulian Anastasov TCP_RTO_MAX / HZ); 24181da177e4SLinus Torvalds break; 24191da177e4SLinus Torvalds 24201da177e4SLinus Torvalds case TCP_WINDOW_CLAMP: 24211da177e4SLinus Torvalds if (!val) { 24221da177e4SLinus Torvalds if (sk->sk_state != TCP_CLOSE) { 24231da177e4SLinus Torvalds err = -EINVAL; 24241da177e4SLinus Torvalds break; 24251da177e4SLinus Torvalds } 24261da177e4SLinus Torvalds tp->window_clamp = 0; 24271da177e4SLinus Torvalds } else 24281da177e4SLinus Torvalds tp->window_clamp = val < SOCK_MIN_RCVBUF / 2 ? 24291da177e4SLinus Torvalds SOCK_MIN_RCVBUF / 2 : val; 24301da177e4SLinus Torvalds break; 24311da177e4SLinus Torvalds 24321da177e4SLinus Torvalds case TCP_QUICKACK: 24331da177e4SLinus Torvalds if (!val) { 2434463c84b9SArnaldo Carvalho de Melo icsk->icsk_ack.pingpong = 1; 24351da177e4SLinus Torvalds } else { 2436463c84b9SArnaldo Carvalho de Melo icsk->icsk_ack.pingpong = 0; 24371da177e4SLinus Torvalds if ((1 << sk->sk_state) & 24381da177e4SLinus Torvalds (TCPF_ESTABLISHED | TCPF_CLOSE_WAIT) && 2439463c84b9SArnaldo Carvalho de Melo inet_csk_ack_scheduled(sk)) { 2440463c84b9SArnaldo Carvalho de Melo icsk->icsk_ack.pending |= ICSK_ACK_PUSHED; 24410e4b4992SChris Leech tcp_cleanup_rbuf(sk, 1); 24421da177e4SLinus Torvalds if (!(val & 1)) 2443463c84b9SArnaldo Carvalho de Melo icsk->icsk_ack.pingpong = 1; 24441da177e4SLinus Torvalds } 24451da177e4SLinus Torvalds } 24461da177e4SLinus Torvalds break; 24471da177e4SLinus Torvalds 2448cfb6eeb4SYOSHIFUJI Hideaki #ifdef CONFIG_TCP_MD5SIG 2449cfb6eeb4SYOSHIFUJI Hideaki case TCP_MD5SIG: 2450cfb6eeb4SYOSHIFUJI Hideaki /* Read the IP->Key mappings from userspace */ 2451cfb6eeb4SYOSHIFUJI Hideaki err = tp->af_specific->md5_parse(sk, optval, optlen); 2452cfb6eeb4SYOSHIFUJI Hideaki break; 2453cfb6eeb4SYOSHIFUJI Hideaki #endif 2454dca43c75SJerry Chu case TCP_USER_TIMEOUT: 2455dca43c75SJerry Chu /* Cap the max timeout in ms TCP will retry/retrans 2456dca43c75SJerry Chu * before giving up and aborting (ETIMEDOUT) a connection. 2457dca43c75SJerry Chu */ 2458dca43c75SJerry Chu icsk->icsk_user_timeout = msecs_to_jiffies(val); 2459dca43c75SJerry Chu break; 24601da177e4SLinus Torvalds default: 24611da177e4SLinus Torvalds err = -ENOPROTOOPT; 24621da177e4SLinus Torvalds break; 24633ff50b79SStephen Hemminger } 24643ff50b79SStephen Hemminger 24651da177e4SLinus Torvalds release_sock(sk); 24661da177e4SLinus Torvalds return err; 24671da177e4SLinus Torvalds } 24681da177e4SLinus Torvalds 24693fdadf7dSDmitry Mishin int tcp_setsockopt(struct sock *sk, int level, int optname, char __user *optval, 2470b7058842SDavid S. Miller unsigned int optlen) 24713fdadf7dSDmitry Mishin { 2472cf533ea5SEric Dumazet const struct inet_connection_sock *icsk = inet_csk(sk); 24733fdadf7dSDmitry Mishin 24743fdadf7dSDmitry Mishin if (level != SOL_TCP) 24753fdadf7dSDmitry Mishin return icsk->icsk_af_ops->setsockopt(sk, level, optname, 24763fdadf7dSDmitry Mishin optval, optlen); 24773fdadf7dSDmitry Mishin return do_tcp_setsockopt(sk, level, optname, optval, optlen); 24783fdadf7dSDmitry Mishin } 24794bc2f18bSEric Dumazet EXPORT_SYMBOL(tcp_setsockopt); 24803fdadf7dSDmitry Mishin 24813fdadf7dSDmitry Mishin #ifdef CONFIG_COMPAT 2482543d9cfeSArnaldo Carvalho de Melo int compat_tcp_setsockopt(struct sock *sk, int level, int optname, 2483b7058842SDavid S. Miller char __user *optval, unsigned int optlen) 24843fdadf7dSDmitry Mishin { 2485dec73ff0SArnaldo Carvalho de Melo if (level != SOL_TCP) 2486dec73ff0SArnaldo Carvalho de Melo return inet_csk_compat_setsockopt(sk, level, optname, 2487dec73ff0SArnaldo Carvalho de Melo optval, optlen); 24883fdadf7dSDmitry Mishin return do_tcp_setsockopt(sk, level, optname, optval, optlen); 24893fdadf7dSDmitry Mishin } 2490543d9cfeSArnaldo Carvalho de Melo EXPORT_SYMBOL(compat_tcp_setsockopt); 24913fdadf7dSDmitry Mishin #endif 24923fdadf7dSDmitry Mishin 24931da177e4SLinus Torvalds /* Return information about state of tcp endpoint in API format. */ 2494cf533ea5SEric Dumazet void tcp_get_info(const struct sock *sk, struct tcp_info *info) 24951da177e4SLinus Torvalds { 2496cf533ea5SEric Dumazet const struct tcp_sock *tp = tcp_sk(sk); 2497463c84b9SArnaldo Carvalho de Melo const struct inet_connection_sock *icsk = inet_csk(sk); 24981da177e4SLinus Torvalds u32 now = tcp_time_stamp; 24991da177e4SLinus Torvalds 25001da177e4SLinus Torvalds memset(info, 0, sizeof(*info)); 25011da177e4SLinus Torvalds 25021da177e4SLinus Torvalds info->tcpi_state = sk->sk_state; 25036687e988SArnaldo Carvalho de Melo info->tcpi_ca_state = icsk->icsk_ca_state; 2504463c84b9SArnaldo Carvalho de Melo info->tcpi_retransmits = icsk->icsk_retransmits; 25056687e988SArnaldo Carvalho de Melo info->tcpi_probes = icsk->icsk_probes_out; 2506463c84b9SArnaldo Carvalho de Melo info->tcpi_backoff = icsk->icsk_backoff; 25071da177e4SLinus Torvalds 25081da177e4SLinus Torvalds if (tp->rx_opt.tstamp_ok) 25091da177e4SLinus Torvalds info->tcpi_options |= TCPI_OPT_TIMESTAMPS; 2510e60402d0SIlpo Järvinen if (tcp_is_sack(tp)) 25111da177e4SLinus Torvalds info->tcpi_options |= TCPI_OPT_SACK; 25121da177e4SLinus Torvalds if (tp->rx_opt.wscale_ok) { 25131da177e4SLinus Torvalds info->tcpi_options |= TCPI_OPT_WSCALE; 25141da177e4SLinus Torvalds info->tcpi_snd_wscale = tp->rx_opt.snd_wscale; 25151da177e4SLinus Torvalds info->tcpi_rcv_wscale = tp->rx_opt.rcv_wscale; 25161da177e4SLinus Torvalds } 25171da177e4SLinus Torvalds 25181da177e4SLinus Torvalds if (tp->ecn_flags & TCP_ECN_OK) 25191da177e4SLinus Torvalds info->tcpi_options |= TCPI_OPT_ECN; 2520b5c5693bSEric Dumazet if (tp->ecn_flags & TCP_ECN_SEEN) 2521b5c5693bSEric Dumazet info->tcpi_options |= TCPI_OPT_ECN_SEEN; 25221da177e4SLinus Torvalds 2523463c84b9SArnaldo Carvalho de Melo info->tcpi_rto = jiffies_to_usecs(icsk->icsk_rto); 2524463c84b9SArnaldo Carvalho de Melo info->tcpi_ato = jiffies_to_usecs(icsk->icsk_ack.ato); 2525c1b4a7e6SDavid S. Miller info->tcpi_snd_mss = tp->mss_cache; 2526463c84b9SArnaldo Carvalho de Melo info->tcpi_rcv_mss = icsk->icsk_ack.rcv_mss; 25271da177e4SLinus Torvalds 25285ee3afbaSRick Jones if (sk->sk_state == TCP_LISTEN) { 25295ee3afbaSRick Jones info->tcpi_unacked = sk->sk_ack_backlog; 25305ee3afbaSRick Jones info->tcpi_sacked = sk->sk_max_ack_backlog; 25315ee3afbaSRick Jones } else { 25321da177e4SLinus Torvalds info->tcpi_unacked = tp->packets_out; 25331da177e4SLinus Torvalds info->tcpi_sacked = tp->sacked_out; 25345ee3afbaSRick Jones } 25351da177e4SLinus Torvalds info->tcpi_lost = tp->lost_out; 25361da177e4SLinus Torvalds info->tcpi_retrans = tp->retrans_out; 25371da177e4SLinus Torvalds info->tcpi_fackets = tp->fackets_out; 25381da177e4SLinus Torvalds 25391da177e4SLinus Torvalds info->tcpi_last_data_sent = jiffies_to_msecs(now - tp->lsndtime); 2540463c84b9SArnaldo Carvalho de Melo info->tcpi_last_data_recv = jiffies_to_msecs(now - icsk->icsk_ack.lrcvtime); 25411da177e4SLinus Torvalds info->tcpi_last_ack_recv = jiffies_to_msecs(now - tp->rcv_tstamp); 25421da177e4SLinus Torvalds 2543d83d8461SArnaldo Carvalho de Melo info->tcpi_pmtu = icsk->icsk_pmtu_cookie; 25441da177e4SLinus Torvalds info->tcpi_rcv_ssthresh = tp->rcv_ssthresh; 25451da177e4SLinus Torvalds info->tcpi_rtt = jiffies_to_usecs(tp->srtt)>>3; 25461da177e4SLinus Torvalds info->tcpi_rttvar = jiffies_to_usecs(tp->mdev)>>2; 25471da177e4SLinus Torvalds info->tcpi_snd_ssthresh = tp->snd_ssthresh; 25481da177e4SLinus Torvalds info->tcpi_snd_cwnd = tp->snd_cwnd; 25491da177e4SLinus Torvalds info->tcpi_advmss = tp->advmss; 25501da177e4SLinus Torvalds info->tcpi_reordering = tp->reordering; 25511da177e4SLinus Torvalds 25521da177e4SLinus Torvalds info->tcpi_rcv_rtt = jiffies_to_usecs(tp->rcv_rtt_est.rtt)>>3; 25531da177e4SLinus Torvalds info->tcpi_rcv_space = tp->rcvq_space.space; 25541da177e4SLinus Torvalds 25551da177e4SLinus Torvalds info->tcpi_total_retrans = tp->total_retrans; 25561da177e4SLinus Torvalds } 25571da177e4SLinus Torvalds EXPORT_SYMBOL_GPL(tcp_get_info); 25581da177e4SLinus Torvalds 25593fdadf7dSDmitry Mishin static int do_tcp_getsockopt(struct sock *sk, int level, 25603fdadf7dSDmitry Mishin int optname, char __user *optval, int __user *optlen) 25611da177e4SLinus Torvalds { 2562295f7324SArnaldo Carvalho de Melo struct inet_connection_sock *icsk = inet_csk(sk); 25631da177e4SLinus Torvalds struct tcp_sock *tp = tcp_sk(sk); 25641da177e4SLinus Torvalds int val, len; 25651da177e4SLinus Torvalds 25661da177e4SLinus Torvalds if (get_user(len, optlen)) 25671da177e4SLinus Torvalds return -EFAULT; 25681da177e4SLinus Torvalds 25691da177e4SLinus Torvalds len = min_t(unsigned int, len, sizeof(int)); 25701da177e4SLinus Torvalds 25711da177e4SLinus Torvalds if (len < 0) 25721da177e4SLinus Torvalds return -EINVAL; 25731da177e4SLinus Torvalds 25741da177e4SLinus Torvalds switch (optname) { 25751da177e4SLinus Torvalds case TCP_MAXSEG: 2576c1b4a7e6SDavid S. Miller val = tp->mss_cache; 25771da177e4SLinus Torvalds if (!val && ((1 << sk->sk_state) & (TCPF_CLOSE | TCPF_LISTEN))) 25781da177e4SLinus Torvalds val = tp->rx_opt.user_mss; 25791da177e4SLinus Torvalds break; 25801da177e4SLinus Torvalds case TCP_NODELAY: 25811da177e4SLinus Torvalds val = !!(tp->nonagle&TCP_NAGLE_OFF); 25821da177e4SLinus Torvalds break; 25831da177e4SLinus Torvalds case TCP_CORK: 25841da177e4SLinus Torvalds val = !!(tp->nonagle&TCP_NAGLE_CORK); 25851da177e4SLinus Torvalds break; 25861da177e4SLinus Torvalds case TCP_KEEPIDLE: 2587df19a626SEric Dumazet val = keepalive_time_when(tp) / HZ; 25881da177e4SLinus Torvalds break; 25891da177e4SLinus Torvalds case TCP_KEEPINTVL: 2590df19a626SEric Dumazet val = keepalive_intvl_when(tp) / HZ; 25911da177e4SLinus Torvalds break; 25921da177e4SLinus Torvalds case TCP_KEEPCNT: 2593df19a626SEric Dumazet val = keepalive_probes(tp); 25941da177e4SLinus Torvalds break; 25951da177e4SLinus Torvalds case TCP_SYNCNT: 2596295f7324SArnaldo Carvalho de Melo val = icsk->icsk_syn_retries ? : sysctl_tcp_syn_retries; 25971da177e4SLinus Torvalds break; 25981da177e4SLinus Torvalds case TCP_LINGER2: 25991da177e4SLinus Torvalds val = tp->linger2; 26001da177e4SLinus Torvalds if (val >= 0) 26011da177e4SLinus Torvalds val = (val ? : sysctl_tcp_fin_timeout) / HZ; 26021da177e4SLinus Torvalds break; 26031da177e4SLinus Torvalds case TCP_DEFER_ACCEPT: 2604b103cf34SJulian Anastasov val = retrans_to_secs(icsk->icsk_accept_queue.rskq_defer_accept, 2605b103cf34SJulian Anastasov TCP_TIMEOUT_INIT / HZ, TCP_RTO_MAX / HZ); 26061da177e4SLinus Torvalds break; 26071da177e4SLinus Torvalds case TCP_WINDOW_CLAMP: 26081da177e4SLinus Torvalds val = tp->window_clamp; 26091da177e4SLinus Torvalds break; 26101da177e4SLinus Torvalds case TCP_INFO: { 26111da177e4SLinus Torvalds struct tcp_info info; 26121da177e4SLinus Torvalds 26131da177e4SLinus Torvalds if (get_user(len, optlen)) 26141da177e4SLinus Torvalds return -EFAULT; 26151da177e4SLinus Torvalds 26161da177e4SLinus Torvalds tcp_get_info(sk, &info); 26171da177e4SLinus Torvalds 26181da177e4SLinus Torvalds len = min_t(unsigned int, len, sizeof(info)); 26191da177e4SLinus Torvalds if (put_user(len, optlen)) 26201da177e4SLinus Torvalds return -EFAULT; 26211da177e4SLinus Torvalds if (copy_to_user(optval, &info, len)) 26221da177e4SLinus Torvalds return -EFAULT; 26231da177e4SLinus Torvalds return 0; 26241da177e4SLinus Torvalds } 26251da177e4SLinus Torvalds case TCP_QUICKACK: 2626295f7324SArnaldo Carvalho de Melo val = !icsk->icsk_ack.pingpong; 26271da177e4SLinus Torvalds break; 26285f8ef48dSStephen Hemminger 26295f8ef48dSStephen Hemminger case TCP_CONGESTION: 26305f8ef48dSStephen Hemminger if (get_user(len, optlen)) 26315f8ef48dSStephen Hemminger return -EFAULT; 26325f8ef48dSStephen Hemminger len = min_t(unsigned int, len, TCP_CA_NAME_MAX); 26335f8ef48dSStephen Hemminger if (put_user(len, optlen)) 26345f8ef48dSStephen Hemminger return -EFAULT; 26356687e988SArnaldo Carvalho de Melo if (copy_to_user(optval, icsk->icsk_ca_ops->name, len)) 26365f8ef48dSStephen Hemminger return -EFAULT; 26375f8ef48dSStephen Hemminger return 0; 2638e56fb50fSWilliam Allen Simpson 2639e56fb50fSWilliam Allen Simpson case TCP_COOKIE_TRANSACTIONS: { 2640e56fb50fSWilliam Allen Simpson struct tcp_cookie_transactions ctd; 2641e56fb50fSWilliam Allen Simpson struct tcp_cookie_values *cvp = tp->cookie_values; 2642e56fb50fSWilliam Allen Simpson 2643e56fb50fSWilliam Allen Simpson if (get_user(len, optlen)) 2644e56fb50fSWilliam Allen Simpson return -EFAULT; 2645e56fb50fSWilliam Allen Simpson if (len < sizeof(ctd)) 2646e56fb50fSWilliam Allen Simpson return -EINVAL; 2647e56fb50fSWilliam Allen Simpson 2648e56fb50fSWilliam Allen Simpson memset(&ctd, 0, sizeof(ctd)); 2649e56fb50fSWilliam Allen Simpson ctd.tcpct_flags = (tp->rx_opt.cookie_in_always ? 2650e56fb50fSWilliam Allen Simpson TCP_COOKIE_IN_ALWAYS : 0) 2651e56fb50fSWilliam Allen Simpson | (tp->rx_opt.cookie_out_never ? 2652e56fb50fSWilliam Allen Simpson TCP_COOKIE_OUT_NEVER : 0); 2653e56fb50fSWilliam Allen Simpson 2654e56fb50fSWilliam Allen Simpson if (cvp != NULL) { 2655e56fb50fSWilliam Allen Simpson ctd.tcpct_flags |= (cvp->s_data_in ? 2656e56fb50fSWilliam Allen Simpson TCP_S_DATA_IN : 0) 2657e56fb50fSWilliam Allen Simpson | (cvp->s_data_out ? 2658e56fb50fSWilliam Allen Simpson TCP_S_DATA_OUT : 0); 2659e56fb50fSWilliam Allen Simpson 2660e56fb50fSWilliam Allen Simpson ctd.tcpct_cookie_desired = cvp->cookie_desired; 2661e56fb50fSWilliam Allen Simpson ctd.tcpct_s_data_desired = cvp->s_data_desired; 2662e56fb50fSWilliam Allen Simpson 2663e56fb50fSWilliam Allen Simpson memcpy(&ctd.tcpct_value[0], &cvp->cookie_pair[0], 2664e56fb50fSWilliam Allen Simpson cvp->cookie_pair_size); 2665e56fb50fSWilliam Allen Simpson ctd.tcpct_used = cvp->cookie_pair_size; 2666e56fb50fSWilliam Allen Simpson } 2667e56fb50fSWilliam Allen Simpson 2668e56fb50fSWilliam Allen Simpson if (put_user(sizeof(ctd), optlen)) 2669e56fb50fSWilliam Allen Simpson return -EFAULT; 2670e56fb50fSWilliam Allen Simpson if (copy_to_user(optval, &ctd, sizeof(ctd))) 2671e56fb50fSWilliam Allen Simpson return -EFAULT; 2672e56fb50fSWilliam Allen Simpson return 0; 2673e56fb50fSWilliam Allen Simpson } 26743c0fef0bSJosh Hunt case TCP_THIN_LINEAR_TIMEOUTS: 26753c0fef0bSJosh Hunt val = tp->thin_lto; 26763c0fef0bSJosh Hunt break; 26773c0fef0bSJosh Hunt case TCP_THIN_DUPACK: 26783c0fef0bSJosh Hunt val = tp->thin_dupack; 26793c0fef0bSJosh Hunt break; 2680dca43c75SJerry Chu 2681*ee995283SPavel Emelyanov case TCP_REPAIR: 2682*ee995283SPavel Emelyanov val = tp->repair; 2683*ee995283SPavel Emelyanov break; 2684*ee995283SPavel Emelyanov 2685*ee995283SPavel Emelyanov case TCP_REPAIR_QUEUE: 2686*ee995283SPavel Emelyanov if (tp->repair) 2687*ee995283SPavel Emelyanov val = tp->repair_queue; 2688*ee995283SPavel Emelyanov else 2689*ee995283SPavel Emelyanov return -EINVAL; 2690*ee995283SPavel Emelyanov break; 2691*ee995283SPavel Emelyanov 2692*ee995283SPavel Emelyanov case TCP_QUEUE_SEQ: 2693*ee995283SPavel Emelyanov if (tp->repair_queue == TCP_SEND_QUEUE) 2694*ee995283SPavel Emelyanov val = tp->write_seq; 2695*ee995283SPavel Emelyanov else if (tp->repair_queue == TCP_RECV_QUEUE) 2696*ee995283SPavel Emelyanov val = tp->rcv_nxt; 2697*ee995283SPavel Emelyanov else 2698*ee995283SPavel Emelyanov return -EINVAL; 2699*ee995283SPavel Emelyanov break; 2700*ee995283SPavel Emelyanov 2701dca43c75SJerry Chu case TCP_USER_TIMEOUT: 2702dca43c75SJerry Chu val = jiffies_to_msecs(icsk->icsk_user_timeout); 2703dca43c75SJerry Chu break; 27041da177e4SLinus Torvalds default: 27051da177e4SLinus Torvalds return -ENOPROTOOPT; 27063ff50b79SStephen Hemminger } 27071da177e4SLinus Torvalds 27081da177e4SLinus Torvalds if (put_user(len, optlen)) 27091da177e4SLinus Torvalds return -EFAULT; 27101da177e4SLinus Torvalds if (copy_to_user(optval, &val, len)) 27111da177e4SLinus Torvalds return -EFAULT; 27121da177e4SLinus Torvalds return 0; 27131da177e4SLinus Torvalds } 27141da177e4SLinus Torvalds 27153fdadf7dSDmitry Mishin int tcp_getsockopt(struct sock *sk, int level, int optname, char __user *optval, 27163fdadf7dSDmitry Mishin int __user *optlen) 27173fdadf7dSDmitry Mishin { 27183fdadf7dSDmitry Mishin struct inet_connection_sock *icsk = inet_csk(sk); 27193fdadf7dSDmitry Mishin 27203fdadf7dSDmitry Mishin if (level != SOL_TCP) 27213fdadf7dSDmitry Mishin return icsk->icsk_af_ops->getsockopt(sk, level, optname, 27223fdadf7dSDmitry Mishin optval, optlen); 27233fdadf7dSDmitry Mishin return do_tcp_getsockopt(sk, level, optname, optval, optlen); 27243fdadf7dSDmitry Mishin } 27254bc2f18bSEric Dumazet EXPORT_SYMBOL(tcp_getsockopt); 27263fdadf7dSDmitry Mishin 27273fdadf7dSDmitry Mishin #ifdef CONFIG_COMPAT 2728543d9cfeSArnaldo Carvalho de Melo int compat_tcp_getsockopt(struct sock *sk, int level, int optname, 2729543d9cfeSArnaldo Carvalho de Melo char __user *optval, int __user *optlen) 27303fdadf7dSDmitry Mishin { 2731dec73ff0SArnaldo Carvalho de Melo if (level != SOL_TCP) 2732dec73ff0SArnaldo Carvalho de Melo return inet_csk_compat_getsockopt(sk, level, optname, 2733dec73ff0SArnaldo Carvalho de Melo optval, optlen); 27343fdadf7dSDmitry Mishin return do_tcp_getsockopt(sk, level, optname, optval, optlen); 27353fdadf7dSDmitry Mishin } 2736543d9cfeSArnaldo Carvalho de Melo EXPORT_SYMBOL(compat_tcp_getsockopt); 27373fdadf7dSDmitry Mishin #endif 27381da177e4SLinus Torvalds 2739c8f44affSMichał Mirosław struct sk_buff *tcp_tso_segment(struct sk_buff *skb, 2740c8f44affSMichał Mirosław netdev_features_t features) 2741f4c50d99SHerbert Xu { 2742f4c50d99SHerbert Xu struct sk_buff *segs = ERR_PTR(-EINVAL); 2743f4c50d99SHerbert Xu struct tcphdr *th; 274495c96174SEric Dumazet unsigned int thlen; 2745f4c50d99SHerbert Xu unsigned int seq; 2746d3bc23e7SAl Viro __be32 delta; 2747f4c50d99SHerbert Xu unsigned int oldlen; 27484e704ee3SHerbert Xu unsigned int mss; 2749f4c50d99SHerbert Xu 2750f4c50d99SHerbert Xu if (!pskb_may_pull(skb, sizeof(*th))) 2751f4c50d99SHerbert Xu goto out; 2752f4c50d99SHerbert Xu 2753aa8223c7SArnaldo Carvalho de Melo th = tcp_hdr(skb); 2754f4c50d99SHerbert Xu thlen = th->doff * 4; 2755f4c50d99SHerbert Xu if (thlen < sizeof(*th)) 2756f4c50d99SHerbert Xu goto out; 2757f4c50d99SHerbert Xu 2758f4c50d99SHerbert Xu if (!pskb_may_pull(skb, thlen)) 2759f4c50d99SHerbert Xu goto out; 2760f4c50d99SHerbert Xu 27610718bcc0SHerbert Xu oldlen = (u16)~skb->len; 2762f4c50d99SHerbert Xu __skb_pull(skb, thlen); 2763f4c50d99SHerbert Xu 27644e704ee3SHerbert Xu mss = skb_shinfo(skb)->gso_size; 27654e704ee3SHerbert Xu if (unlikely(skb->len <= mss)) 27664e704ee3SHerbert Xu goto out; 27674e704ee3SHerbert Xu 27683820c3f3SHerbert Xu if (skb_gso_ok(skb, features | NETIF_F_GSO_ROBUST)) { 27693820c3f3SHerbert Xu /* Packet is from an untrusted source, reset gso_segs. */ 2770bbcf467dSHerbert Xu int type = skb_shinfo(skb)->gso_type; 27713820c3f3SHerbert Xu 2772bbcf467dSHerbert Xu if (unlikely(type & 2773bbcf467dSHerbert Xu ~(SKB_GSO_TCPV4 | 2774bbcf467dSHerbert Xu SKB_GSO_DODGY | 2775bbcf467dSHerbert Xu SKB_GSO_TCP_ECN | 2776bbcf467dSHerbert Xu SKB_GSO_TCPV6 | 2777bbcf467dSHerbert Xu 0) || 2778bbcf467dSHerbert Xu !(type & (SKB_GSO_TCPV4 | SKB_GSO_TCPV6)))) 2779bbcf467dSHerbert Xu goto out; 2780bbcf467dSHerbert Xu 2781172589ccSIlpo Järvinen skb_shinfo(skb)->gso_segs = DIV_ROUND_UP(skb->len, mss); 27823820c3f3SHerbert Xu 27833820c3f3SHerbert Xu segs = NULL; 27843820c3f3SHerbert Xu goto out; 27853820c3f3SHerbert Xu } 27863820c3f3SHerbert Xu 2787576a30ebSHerbert Xu segs = skb_segment(skb, features); 2788f4c50d99SHerbert Xu if (IS_ERR(segs)) 2789f4c50d99SHerbert Xu goto out; 2790f4c50d99SHerbert Xu 27914e704ee3SHerbert Xu delta = htonl(oldlen + (thlen + mss)); 2792f4c50d99SHerbert Xu 2793f4c50d99SHerbert Xu skb = segs; 2794aa8223c7SArnaldo Carvalho de Melo th = tcp_hdr(skb); 2795f4c50d99SHerbert Xu seq = ntohl(th->seq); 2796f4c50d99SHerbert Xu 2797f4c50d99SHerbert Xu do { 2798f4c50d99SHerbert Xu th->fin = th->psh = 0; 2799f4c50d99SHerbert Xu 2800d3bc23e7SAl Viro th->check = ~csum_fold((__force __wsum)((__force u32)th->check + 2801d3bc23e7SAl Viro (__force u32)delta)); 280284fa7933SPatrick McHardy if (skb->ip_summed != CHECKSUM_PARTIAL) 28039c70220bSArnaldo Carvalho de Melo th->check = 28049c70220bSArnaldo Carvalho de Melo csum_fold(csum_partial(skb_transport_header(skb), 28059c70220bSArnaldo Carvalho de Melo thlen, skb->csum)); 2806f4c50d99SHerbert Xu 28074e704ee3SHerbert Xu seq += mss; 2808f4c50d99SHerbert Xu skb = skb->next; 2809aa8223c7SArnaldo Carvalho de Melo th = tcp_hdr(skb); 2810f4c50d99SHerbert Xu 2811f4c50d99SHerbert Xu th->seq = htonl(seq); 2812f4c50d99SHerbert Xu th->cwr = 0; 2813f4c50d99SHerbert Xu } while (skb->next); 2814f4c50d99SHerbert Xu 281527a884dcSArnaldo Carvalho de Melo delta = htonl(oldlen + (skb->tail - skb->transport_header) + 28169c70220bSArnaldo Carvalho de Melo skb->data_len); 2817d3bc23e7SAl Viro th->check = ~csum_fold((__force __wsum)((__force u32)th->check + 2818d3bc23e7SAl Viro (__force u32)delta)); 281984fa7933SPatrick McHardy if (skb->ip_summed != CHECKSUM_PARTIAL) 28209c70220bSArnaldo Carvalho de Melo th->check = csum_fold(csum_partial(skb_transport_header(skb), 28219c70220bSArnaldo Carvalho de Melo thlen, skb->csum)); 2822f4c50d99SHerbert Xu 2823f4c50d99SHerbert Xu out: 2824f4c50d99SHerbert Xu return segs; 2825f4c50d99SHerbert Xu } 2826adcfc7d0SHerbert Xu EXPORT_SYMBOL(tcp_tso_segment); 2827f4c50d99SHerbert Xu 2828bf296b12SHerbert Xu struct sk_buff **tcp_gro_receive(struct sk_buff **head, struct sk_buff *skb) 2829bf296b12SHerbert Xu { 2830bf296b12SHerbert Xu struct sk_buff **pp = NULL; 2831bf296b12SHerbert Xu struct sk_buff *p; 2832bf296b12SHerbert Xu struct tcphdr *th; 2833bf296b12SHerbert Xu struct tcphdr *th2; 2834a0a69a01SHerbert Xu unsigned int len; 2835bf296b12SHerbert Xu unsigned int thlen; 28360eae88f3SEric Dumazet __be32 flags; 2837bf296b12SHerbert Xu unsigned int mss = 1; 2838a5b1cf28SHerbert Xu unsigned int hlen; 2839a5b1cf28SHerbert Xu unsigned int off; 2840bf296b12SHerbert Xu int flush = 1; 2841aa6320d3SHerbert Xu int i; 2842bf296b12SHerbert Xu 2843a5b1cf28SHerbert Xu off = skb_gro_offset(skb); 2844a5b1cf28SHerbert Xu hlen = off + sizeof(*th); 2845a5b1cf28SHerbert Xu th = skb_gro_header_fast(skb, off); 2846a5b1cf28SHerbert Xu if (skb_gro_header_hard(skb, hlen)) { 2847a5b1cf28SHerbert Xu th = skb_gro_header_slow(skb, hlen, off); 284886911732SHerbert Xu if (unlikely(!th)) 2849bf296b12SHerbert Xu goto out; 2850a5b1cf28SHerbert Xu } 2851bf296b12SHerbert Xu 2852bf296b12SHerbert Xu thlen = th->doff * 4; 2853bf296b12SHerbert Xu if (thlen < sizeof(*th)) 2854bf296b12SHerbert Xu goto out; 2855bf296b12SHerbert Xu 2856a5b1cf28SHerbert Xu hlen = off + thlen; 2857a5b1cf28SHerbert Xu if (skb_gro_header_hard(skb, hlen)) { 2858a5b1cf28SHerbert Xu th = skb_gro_header_slow(skb, hlen, off); 285986911732SHerbert Xu if (unlikely(!th)) 2860bf296b12SHerbert Xu goto out; 2861a5b1cf28SHerbert Xu } 2862bf296b12SHerbert Xu 286386911732SHerbert Xu skb_gro_pull(skb, thlen); 2864bf296b12SHerbert Xu 2865a0a69a01SHerbert Xu len = skb_gro_len(skb); 2866bf296b12SHerbert Xu flags = tcp_flag_word(th); 2867bf296b12SHerbert Xu 2868bf296b12SHerbert Xu for (; (p = *head); head = &p->next) { 2869bf296b12SHerbert Xu if (!NAPI_GRO_CB(p)->same_flow) 2870bf296b12SHerbert Xu continue; 2871bf296b12SHerbert Xu 2872bf296b12SHerbert Xu th2 = tcp_hdr(p); 2873bf296b12SHerbert Xu 2874745898eaSHerbert Xu if (*(u32 *)&th->source ^ *(u32 *)&th2->source) { 2875bf296b12SHerbert Xu NAPI_GRO_CB(p)->same_flow = 0; 2876bf296b12SHerbert Xu continue; 2877bf296b12SHerbert Xu } 2878bf296b12SHerbert Xu 2879bf296b12SHerbert Xu goto found; 2880bf296b12SHerbert Xu } 2881bf296b12SHerbert Xu 2882bf296b12SHerbert Xu goto out_check_final; 2883bf296b12SHerbert Xu 2884bf296b12SHerbert Xu found: 2885bf296b12SHerbert Xu flush = NAPI_GRO_CB(p)->flush; 28860eae88f3SEric Dumazet flush |= (__force int)(flags & TCP_FLAG_CWR); 28870eae88f3SEric Dumazet flush |= (__force int)((flags ^ tcp_flag_word(th2)) & 28880eae88f3SEric Dumazet ~(TCP_FLAG_CWR | TCP_FLAG_FIN | TCP_FLAG_PSH)); 28890eae88f3SEric Dumazet flush |= (__force int)(th->ack_seq ^ th2->ack_seq); 2890a2a804cdSHerbert Xu for (i = sizeof(*th); i < thlen; i += 4) 2891aa6320d3SHerbert Xu flush |= *(u32 *)((u8 *)th + i) ^ 2892aa6320d3SHerbert Xu *(u32 *)((u8 *)th2 + i); 2893bf296b12SHerbert Xu 2894b530256dSHerbert Xu mss = skb_shinfo(p)->gso_size; 2895bf296b12SHerbert Xu 289630a3ae30SHerbert Xu flush |= (len - 1) >= mss; 2897aa6320d3SHerbert Xu flush |= (ntohl(th2->seq) + skb_gro_len(p)) ^ ntohl(th->seq); 2898bf296b12SHerbert Xu 2899bf296b12SHerbert Xu if (flush || skb_gro_receive(head, skb)) { 2900bf296b12SHerbert Xu mss = 1; 2901bf296b12SHerbert Xu goto out_check_final; 2902bf296b12SHerbert Xu } 2903bf296b12SHerbert Xu 2904bf296b12SHerbert Xu p = *head; 2905bf296b12SHerbert Xu th2 = tcp_hdr(p); 2906bf296b12SHerbert Xu tcp_flag_word(th2) |= flags & (TCP_FLAG_FIN | TCP_FLAG_PSH); 2907bf296b12SHerbert Xu 2908bf296b12SHerbert Xu out_check_final: 2909a0a69a01SHerbert Xu flush = len < mss; 29100eae88f3SEric Dumazet flush |= (__force int)(flags & (TCP_FLAG_URG | TCP_FLAG_PSH | 29110eae88f3SEric Dumazet TCP_FLAG_RST | TCP_FLAG_SYN | 29120eae88f3SEric Dumazet TCP_FLAG_FIN)); 2913bf296b12SHerbert Xu 2914bf296b12SHerbert Xu if (p && (!NAPI_GRO_CB(skb)->same_flow || flush)) 2915bf296b12SHerbert Xu pp = head; 2916bf296b12SHerbert Xu 2917bf296b12SHerbert Xu out: 2918bf296b12SHerbert Xu NAPI_GRO_CB(skb)->flush |= flush; 2919bf296b12SHerbert Xu 2920bf296b12SHerbert Xu return pp; 2921bf296b12SHerbert Xu } 2922684f2176SHerbert Xu EXPORT_SYMBOL(tcp_gro_receive); 2923bf296b12SHerbert Xu 2924bf296b12SHerbert Xu int tcp_gro_complete(struct sk_buff *skb) 2925bf296b12SHerbert Xu { 2926bf296b12SHerbert Xu struct tcphdr *th = tcp_hdr(skb); 2927bf296b12SHerbert Xu 2928bf296b12SHerbert Xu skb->csum_start = skb_transport_header(skb) - skb->head; 2929bf296b12SHerbert Xu skb->csum_offset = offsetof(struct tcphdr, check); 2930bf296b12SHerbert Xu skb->ip_summed = CHECKSUM_PARTIAL; 2931bf296b12SHerbert Xu 2932bf296b12SHerbert Xu skb_shinfo(skb)->gso_segs = NAPI_GRO_CB(skb)->count; 2933bf296b12SHerbert Xu 2934bf296b12SHerbert Xu if (th->cwr) 2935bf296b12SHerbert Xu skb_shinfo(skb)->gso_type |= SKB_GSO_TCP_ECN; 2936bf296b12SHerbert Xu 2937bf296b12SHerbert Xu return 0; 2938bf296b12SHerbert Xu } 2939684f2176SHerbert Xu EXPORT_SYMBOL(tcp_gro_complete); 2940bf296b12SHerbert Xu 2941cfb6eeb4SYOSHIFUJI Hideaki #ifdef CONFIG_TCP_MD5SIG 2942cfb6eeb4SYOSHIFUJI Hideaki static unsigned long tcp_md5sig_users; 2943765cf997SEric Dumazet static struct tcp_md5sig_pool __percpu *tcp_md5sig_pool; 2944cfb6eeb4SYOSHIFUJI Hideaki static DEFINE_SPINLOCK(tcp_md5sig_pool_lock); 2945cfb6eeb4SYOSHIFUJI Hideaki 2946765cf997SEric Dumazet static void __tcp_free_md5sig_pool(struct tcp_md5sig_pool __percpu *pool) 2947cfb6eeb4SYOSHIFUJI Hideaki { 2948cfb6eeb4SYOSHIFUJI Hideaki int cpu; 2949765cf997SEric Dumazet 2950cfb6eeb4SYOSHIFUJI Hideaki for_each_possible_cpu(cpu) { 2951765cf997SEric Dumazet struct tcp_md5sig_pool *p = per_cpu_ptr(pool, cpu); 2952765cf997SEric Dumazet 2953cfb6eeb4SYOSHIFUJI Hideaki if (p->md5_desc.tfm) 2954cfb6eeb4SYOSHIFUJI Hideaki crypto_free_hash(p->md5_desc.tfm); 2955cfb6eeb4SYOSHIFUJI Hideaki } 2956cfb6eeb4SYOSHIFUJI Hideaki free_percpu(pool); 2957cfb6eeb4SYOSHIFUJI Hideaki } 2958cfb6eeb4SYOSHIFUJI Hideaki 2959cfb6eeb4SYOSHIFUJI Hideaki void tcp_free_md5sig_pool(void) 2960cfb6eeb4SYOSHIFUJI Hideaki { 2961765cf997SEric Dumazet struct tcp_md5sig_pool __percpu *pool = NULL; 2962cfb6eeb4SYOSHIFUJI Hideaki 29632c4f6219SDavid S. Miller spin_lock_bh(&tcp_md5sig_pool_lock); 2964cfb6eeb4SYOSHIFUJI Hideaki if (--tcp_md5sig_users == 0) { 2965cfb6eeb4SYOSHIFUJI Hideaki pool = tcp_md5sig_pool; 2966cfb6eeb4SYOSHIFUJI Hideaki tcp_md5sig_pool = NULL; 2967cfb6eeb4SYOSHIFUJI Hideaki } 29682c4f6219SDavid S. Miller spin_unlock_bh(&tcp_md5sig_pool_lock); 2969cfb6eeb4SYOSHIFUJI Hideaki if (pool) 2970cfb6eeb4SYOSHIFUJI Hideaki __tcp_free_md5sig_pool(pool); 2971cfb6eeb4SYOSHIFUJI Hideaki } 2972cfb6eeb4SYOSHIFUJI Hideaki EXPORT_SYMBOL(tcp_free_md5sig_pool); 2973cfb6eeb4SYOSHIFUJI Hideaki 2974765cf997SEric Dumazet static struct tcp_md5sig_pool __percpu * 29757d720c3eSTejun Heo __tcp_alloc_md5sig_pool(struct sock *sk) 2976cfb6eeb4SYOSHIFUJI Hideaki { 2977cfb6eeb4SYOSHIFUJI Hideaki int cpu; 2978765cf997SEric Dumazet struct tcp_md5sig_pool __percpu *pool; 2979cfb6eeb4SYOSHIFUJI Hideaki 2980765cf997SEric Dumazet pool = alloc_percpu(struct tcp_md5sig_pool); 2981cfb6eeb4SYOSHIFUJI Hideaki if (!pool) 2982cfb6eeb4SYOSHIFUJI Hideaki return NULL; 2983cfb6eeb4SYOSHIFUJI Hideaki 2984cfb6eeb4SYOSHIFUJI Hideaki for_each_possible_cpu(cpu) { 2985cfb6eeb4SYOSHIFUJI Hideaki struct crypto_hash *hash; 2986cfb6eeb4SYOSHIFUJI Hideaki 2987cfb6eeb4SYOSHIFUJI Hideaki hash = crypto_alloc_hash("md5", 0, CRYPTO_ALG_ASYNC); 2988cfb6eeb4SYOSHIFUJI Hideaki if (!hash || IS_ERR(hash)) 2989cfb6eeb4SYOSHIFUJI Hideaki goto out_free; 2990cfb6eeb4SYOSHIFUJI Hideaki 2991765cf997SEric Dumazet per_cpu_ptr(pool, cpu)->md5_desc.tfm = hash; 2992cfb6eeb4SYOSHIFUJI Hideaki } 2993cfb6eeb4SYOSHIFUJI Hideaki return pool; 2994cfb6eeb4SYOSHIFUJI Hideaki out_free: 2995cfb6eeb4SYOSHIFUJI Hideaki __tcp_free_md5sig_pool(pool); 2996cfb6eeb4SYOSHIFUJI Hideaki return NULL; 2997cfb6eeb4SYOSHIFUJI Hideaki } 2998cfb6eeb4SYOSHIFUJI Hideaki 2999765cf997SEric Dumazet struct tcp_md5sig_pool __percpu *tcp_alloc_md5sig_pool(struct sock *sk) 3000cfb6eeb4SYOSHIFUJI Hideaki { 3001765cf997SEric Dumazet struct tcp_md5sig_pool __percpu *pool; 3002cfb6eeb4SYOSHIFUJI Hideaki int alloc = 0; 3003cfb6eeb4SYOSHIFUJI Hideaki 3004cfb6eeb4SYOSHIFUJI Hideaki retry: 30052c4f6219SDavid S. Miller spin_lock_bh(&tcp_md5sig_pool_lock); 3006cfb6eeb4SYOSHIFUJI Hideaki pool = tcp_md5sig_pool; 3007cfb6eeb4SYOSHIFUJI Hideaki if (tcp_md5sig_users++ == 0) { 3008cfb6eeb4SYOSHIFUJI Hideaki alloc = 1; 30092c4f6219SDavid S. Miller spin_unlock_bh(&tcp_md5sig_pool_lock); 3010cfb6eeb4SYOSHIFUJI Hideaki } else if (!pool) { 3011cfb6eeb4SYOSHIFUJI Hideaki tcp_md5sig_users--; 30122c4f6219SDavid S. Miller spin_unlock_bh(&tcp_md5sig_pool_lock); 3013cfb6eeb4SYOSHIFUJI Hideaki cpu_relax(); 3014cfb6eeb4SYOSHIFUJI Hideaki goto retry; 3015cfb6eeb4SYOSHIFUJI Hideaki } else 30162c4f6219SDavid S. Miller spin_unlock_bh(&tcp_md5sig_pool_lock); 3017cfb6eeb4SYOSHIFUJI Hideaki 3018cfb6eeb4SYOSHIFUJI Hideaki if (alloc) { 3019cfb6eeb4SYOSHIFUJI Hideaki /* we cannot hold spinlock here because this may sleep. */ 3020765cf997SEric Dumazet struct tcp_md5sig_pool __percpu *p; 30217d720c3eSTejun Heo 30227d720c3eSTejun Heo p = __tcp_alloc_md5sig_pool(sk); 30232c4f6219SDavid S. Miller spin_lock_bh(&tcp_md5sig_pool_lock); 3024cfb6eeb4SYOSHIFUJI Hideaki if (!p) { 3025cfb6eeb4SYOSHIFUJI Hideaki tcp_md5sig_users--; 30262c4f6219SDavid S. Miller spin_unlock_bh(&tcp_md5sig_pool_lock); 3027cfb6eeb4SYOSHIFUJI Hideaki return NULL; 3028cfb6eeb4SYOSHIFUJI Hideaki } 3029cfb6eeb4SYOSHIFUJI Hideaki pool = tcp_md5sig_pool; 3030cfb6eeb4SYOSHIFUJI Hideaki if (pool) { 3031cfb6eeb4SYOSHIFUJI Hideaki /* oops, it has already been assigned. */ 30322c4f6219SDavid S. Miller spin_unlock_bh(&tcp_md5sig_pool_lock); 3033cfb6eeb4SYOSHIFUJI Hideaki __tcp_free_md5sig_pool(p); 3034cfb6eeb4SYOSHIFUJI Hideaki } else { 3035cfb6eeb4SYOSHIFUJI Hideaki tcp_md5sig_pool = pool = p; 30362c4f6219SDavid S. Miller spin_unlock_bh(&tcp_md5sig_pool_lock); 3037cfb6eeb4SYOSHIFUJI Hideaki } 3038cfb6eeb4SYOSHIFUJI Hideaki } 3039cfb6eeb4SYOSHIFUJI Hideaki return pool; 3040cfb6eeb4SYOSHIFUJI Hideaki } 3041cfb6eeb4SYOSHIFUJI Hideaki EXPORT_SYMBOL(tcp_alloc_md5sig_pool); 3042cfb6eeb4SYOSHIFUJI Hideaki 304335790c04SEric Dumazet 304435790c04SEric Dumazet /** 304535790c04SEric Dumazet * tcp_get_md5sig_pool - get md5sig_pool for this user 304635790c04SEric Dumazet * 304735790c04SEric Dumazet * We use percpu structure, so if we succeed, we exit with preemption 304835790c04SEric Dumazet * and BH disabled, to make sure another thread or softirq handling 304935790c04SEric Dumazet * wont try to get same context. 305035790c04SEric Dumazet */ 305135790c04SEric Dumazet struct tcp_md5sig_pool *tcp_get_md5sig_pool(void) 3052cfb6eeb4SYOSHIFUJI Hideaki { 3053765cf997SEric Dumazet struct tcp_md5sig_pool __percpu *p; 305435790c04SEric Dumazet 305535790c04SEric Dumazet local_bh_disable(); 305635790c04SEric Dumazet 305735790c04SEric Dumazet spin_lock(&tcp_md5sig_pool_lock); 3058cfb6eeb4SYOSHIFUJI Hideaki p = tcp_md5sig_pool; 3059cfb6eeb4SYOSHIFUJI Hideaki if (p) 3060cfb6eeb4SYOSHIFUJI Hideaki tcp_md5sig_users++; 306135790c04SEric Dumazet spin_unlock(&tcp_md5sig_pool_lock); 306235790c04SEric Dumazet 306335790c04SEric Dumazet if (p) 3064765cf997SEric Dumazet return this_cpu_ptr(p); 306535790c04SEric Dumazet 306635790c04SEric Dumazet local_bh_enable(); 306735790c04SEric Dumazet return NULL; 3068cfb6eeb4SYOSHIFUJI Hideaki } 306935790c04SEric Dumazet EXPORT_SYMBOL(tcp_get_md5sig_pool); 3070cfb6eeb4SYOSHIFUJI Hideaki 307135790c04SEric Dumazet void tcp_put_md5sig_pool(void) 30726931ba7cSDavid S. Miller { 307335790c04SEric Dumazet local_bh_enable(); 30746931ba7cSDavid S. Miller tcp_free_md5sig_pool(); 3075cfb6eeb4SYOSHIFUJI Hideaki } 307635790c04SEric Dumazet EXPORT_SYMBOL(tcp_put_md5sig_pool); 307749a72dfbSAdam Langley 307849a72dfbSAdam Langley int tcp_md5_hash_header(struct tcp_md5sig_pool *hp, 3079ca35a0efSEric Dumazet const struct tcphdr *th) 308049a72dfbSAdam Langley { 308149a72dfbSAdam Langley struct scatterlist sg; 3082ca35a0efSEric Dumazet struct tcphdr hdr; 308349a72dfbSAdam Langley int err; 308449a72dfbSAdam Langley 3085ca35a0efSEric Dumazet /* We are not allowed to change tcphdr, make a local copy */ 3086ca35a0efSEric Dumazet memcpy(&hdr, th, sizeof(hdr)); 3087ca35a0efSEric Dumazet hdr.check = 0; 3088ca35a0efSEric Dumazet 308949a72dfbSAdam Langley /* options aren't included in the hash */ 3090ca35a0efSEric Dumazet sg_init_one(&sg, &hdr, sizeof(hdr)); 3091ca35a0efSEric Dumazet err = crypto_hash_update(&hp->md5_desc, &sg, sizeof(hdr)); 309249a72dfbSAdam Langley return err; 309349a72dfbSAdam Langley } 309449a72dfbSAdam Langley EXPORT_SYMBOL(tcp_md5_hash_header); 309549a72dfbSAdam Langley 309649a72dfbSAdam Langley int tcp_md5_hash_skb_data(struct tcp_md5sig_pool *hp, 3097cf533ea5SEric Dumazet const struct sk_buff *skb, unsigned int header_len) 309849a72dfbSAdam Langley { 309949a72dfbSAdam Langley struct scatterlist sg; 310049a72dfbSAdam Langley const struct tcphdr *tp = tcp_hdr(skb); 310149a72dfbSAdam Langley struct hash_desc *desc = &hp->md5_desc; 310295c96174SEric Dumazet unsigned int i; 310395c96174SEric Dumazet const unsigned int head_data_len = skb_headlen(skb) > header_len ? 310449a72dfbSAdam Langley skb_headlen(skb) - header_len : 0; 310549a72dfbSAdam Langley const struct skb_shared_info *shi = skb_shinfo(skb); 3106d7fd1b57SEric Dumazet struct sk_buff *frag_iter; 310749a72dfbSAdam Langley 310849a72dfbSAdam Langley sg_init_table(&sg, 1); 310949a72dfbSAdam Langley 311049a72dfbSAdam Langley sg_set_buf(&sg, ((u8 *) tp) + header_len, head_data_len); 311149a72dfbSAdam Langley if (crypto_hash_update(desc, &sg, head_data_len)) 311249a72dfbSAdam Langley return 1; 311349a72dfbSAdam Langley 311449a72dfbSAdam Langley for (i = 0; i < shi->nr_frags; ++i) { 311549a72dfbSAdam Langley const struct skb_frag_struct *f = &shi->frags[i]; 3116aff65da0SIan Campbell struct page *page = skb_frag_page(f); 31179e903e08SEric Dumazet sg_set_page(&sg, page, skb_frag_size(f), f->page_offset); 31189e903e08SEric Dumazet if (crypto_hash_update(desc, &sg, skb_frag_size(f))) 311949a72dfbSAdam Langley return 1; 312049a72dfbSAdam Langley } 312149a72dfbSAdam Langley 3122d7fd1b57SEric Dumazet skb_walk_frags(skb, frag_iter) 3123d7fd1b57SEric Dumazet if (tcp_md5_hash_skb_data(hp, frag_iter, 0)) 3124d7fd1b57SEric Dumazet return 1; 3125d7fd1b57SEric Dumazet 312649a72dfbSAdam Langley return 0; 312749a72dfbSAdam Langley } 312849a72dfbSAdam Langley EXPORT_SYMBOL(tcp_md5_hash_skb_data); 312949a72dfbSAdam Langley 3130cf533ea5SEric Dumazet int tcp_md5_hash_key(struct tcp_md5sig_pool *hp, const struct tcp_md5sig_key *key) 313149a72dfbSAdam Langley { 313249a72dfbSAdam Langley struct scatterlist sg; 313349a72dfbSAdam Langley 313449a72dfbSAdam Langley sg_init_one(&sg, key->key, key->keylen); 313549a72dfbSAdam Langley return crypto_hash_update(&hp->md5_desc, &sg, key->keylen); 313649a72dfbSAdam Langley } 313749a72dfbSAdam Langley EXPORT_SYMBOL(tcp_md5_hash_key); 313849a72dfbSAdam Langley 3139cfb6eeb4SYOSHIFUJI Hideaki #endif 3140cfb6eeb4SYOSHIFUJI Hideaki 3141da5c78c8SWilliam Allen Simpson /** 3142da5c78c8SWilliam Allen Simpson * Each Responder maintains up to two secret values concurrently for 3143da5c78c8SWilliam Allen Simpson * efficient secret rollover. Each secret value has 4 states: 3144da5c78c8SWilliam Allen Simpson * 3145da5c78c8SWilliam Allen Simpson * Generating. (tcp_secret_generating != tcp_secret_primary) 3146da5c78c8SWilliam Allen Simpson * Generates new Responder-Cookies, but not yet used for primary 3147da5c78c8SWilliam Allen Simpson * verification. This is a short-term state, typically lasting only 3148da5c78c8SWilliam Allen Simpson * one round trip time (RTT). 3149da5c78c8SWilliam Allen Simpson * 3150da5c78c8SWilliam Allen Simpson * Primary. (tcp_secret_generating == tcp_secret_primary) 3151da5c78c8SWilliam Allen Simpson * Used both for generation and primary verification. 3152da5c78c8SWilliam Allen Simpson * 3153da5c78c8SWilliam Allen Simpson * Retiring. (tcp_secret_retiring != tcp_secret_secondary) 3154da5c78c8SWilliam Allen Simpson * Used for verification, until the first failure that can be 3155da5c78c8SWilliam Allen Simpson * verified by the newer Generating secret. At that time, this 3156da5c78c8SWilliam Allen Simpson * cookie's state is changed to Secondary, and the Generating 3157da5c78c8SWilliam Allen Simpson * cookie's state is changed to Primary. This is a short-term state, 3158da5c78c8SWilliam Allen Simpson * typically lasting only one round trip time (RTT). 3159da5c78c8SWilliam Allen Simpson * 3160da5c78c8SWilliam Allen Simpson * Secondary. (tcp_secret_retiring == tcp_secret_secondary) 3161da5c78c8SWilliam Allen Simpson * Used for secondary verification, after primary verification 3162da5c78c8SWilliam Allen Simpson * failures. This state lasts no more than twice the Maximum Segment 3163da5c78c8SWilliam Allen Simpson * Lifetime (2MSL). Then, the secret is discarded. 3164da5c78c8SWilliam Allen Simpson */ 3165da5c78c8SWilliam Allen Simpson struct tcp_cookie_secret { 3166da5c78c8SWilliam Allen Simpson /* The secret is divided into two parts. The digest part is the 3167da5c78c8SWilliam Allen Simpson * equivalent of previously hashing a secret and saving the state, 3168da5c78c8SWilliam Allen Simpson * and serves as an initialization vector (IV). The message part 3169da5c78c8SWilliam Allen Simpson * serves as the trailing secret. 3170da5c78c8SWilliam Allen Simpson */ 3171da5c78c8SWilliam Allen Simpson u32 secrets[COOKIE_WORKSPACE_WORDS]; 3172da5c78c8SWilliam Allen Simpson unsigned long expires; 3173da5c78c8SWilliam Allen Simpson }; 3174da5c78c8SWilliam Allen Simpson 3175da5c78c8SWilliam Allen Simpson #define TCP_SECRET_1MSL (HZ * TCP_PAWS_MSL) 3176da5c78c8SWilliam Allen Simpson #define TCP_SECRET_2MSL (HZ * TCP_PAWS_MSL * 2) 3177da5c78c8SWilliam Allen Simpson #define TCP_SECRET_LIFE (HZ * 600) 3178da5c78c8SWilliam Allen Simpson 3179da5c78c8SWilliam Allen Simpson static struct tcp_cookie_secret tcp_secret_one; 3180da5c78c8SWilliam Allen Simpson static struct tcp_cookie_secret tcp_secret_two; 3181da5c78c8SWilliam Allen Simpson 3182da5c78c8SWilliam Allen Simpson /* Essentially a circular list, without dynamic allocation. */ 3183da5c78c8SWilliam Allen Simpson static struct tcp_cookie_secret *tcp_secret_generating; 3184da5c78c8SWilliam Allen Simpson static struct tcp_cookie_secret *tcp_secret_primary; 3185da5c78c8SWilliam Allen Simpson static struct tcp_cookie_secret *tcp_secret_retiring; 3186da5c78c8SWilliam Allen Simpson static struct tcp_cookie_secret *tcp_secret_secondary; 3187da5c78c8SWilliam Allen Simpson 3188da5c78c8SWilliam Allen Simpson static DEFINE_SPINLOCK(tcp_secret_locker); 3189da5c78c8SWilliam Allen Simpson 3190da5c78c8SWilliam Allen Simpson /* Select a pseudo-random word in the cookie workspace. 3191da5c78c8SWilliam Allen Simpson */ 3192da5c78c8SWilliam Allen Simpson static inline u32 tcp_cookie_work(const u32 *ws, const int n) 3193da5c78c8SWilliam Allen Simpson { 3194da5c78c8SWilliam Allen Simpson return ws[COOKIE_DIGEST_WORDS + ((COOKIE_MESSAGE_WORDS-1) & ws[n])]; 3195da5c78c8SWilliam Allen Simpson } 3196da5c78c8SWilliam Allen Simpson 3197da5c78c8SWilliam Allen Simpson /* Fill bakery[COOKIE_WORKSPACE_WORDS] with generator, updating as needed. 3198da5c78c8SWilliam Allen Simpson * Called in softirq context. 3199da5c78c8SWilliam Allen Simpson * Returns: 0 for success. 3200da5c78c8SWilliam Allen Simpson */ 3201da5c78c8SWilliam Allen Simpson int tcp_cookie_generator(u32 *bakery) 3202da5c78c8SWilliam Allen Simpson { 3203da5c78c8SWilliam Allen Simpson unsigned long jiffy = jiffies; 3204da5c78c8SWilliam Allen Simpson 3205da5c78c8SWilliam Allen Simpson if (unlikely(time_after_eq(jiffy, tcp_secret_generating->expires))) { 3206da5c78c8SWilliam Allen Simpson spin_lock_bh(&tcp_secret_locker); 3207da5c78c8SWilliam Allen Simpson if (!time_after_eq(jiffy, tcp_secret_generating->expires)) { 3208da5c78c8SWilliam Allen Simpson /* refreshed by another */ 3209da5c78c8SWilliam Allen Simpson memcpy(bakery, 3210da5c78c8SWilliam Allen Simpson &tcp_secret_generating->secrets[0], 3211da5c78c8SWilliam Allen Simpson COOKIE_WORKSPACE_WORDS); 3212da5c78c8SWilliam Allen Simpson } else { 3213da5c78c8SWilliam Allen Simpson /* still needs refreshing */ 3214da5c78c8SWilliam Allen Simpson get_random_bytes(bakery, COOKIE_WORKSPACE_WORDS); 3215da5c78c8SWilliam Allen Simpson 3216da5c78c8SWilliam Allen Simpson /* The first time, paranoia assumes that the 3217da5c78c8SWilliam Allen Simpson * randomization function isn't as strong. But, 3218da5c78c8SWilliam Allen Simpson * this secret initialization is delayed until 3219da5c78c8SWilliam Allen Simpson * the last possible moment (packet arrival). 3220da5c78c8SWilliam Allen Simpson * Although that time is observable, it is 3221da5c78c8SWilliam Allen Simpson * unpredictably variable. Mash in the most 3222da5c78c8SWilliam Allen Simpson * volatile clock bits available, and expire the 3223da5c78c8SWilliam Allen Simpson * secret extra quickly. 3224da5c78c8SWilliam Allen Simpson */ 3225da5c78c8SWilliam Allen Simpson if (unlikely(tcp_secret_primary->expires == 3226da5c78c8SWilliam Allen Simpson tcp_secret_secondary->expires)) { 3227da5c78c8SWilliam Allen Simpson struct timespec tv; 3228da5c78c8SWilliam Allen Simpson 3229da5c78c8SWilliam Allen Simpson getnstimeofday(&tv); 3230da5c78c8SWilliam Allen Simpson bakery[COOKIE_DIGEST_WORDS+0] ^= 3231da5c78c8SWilliam Allen Simpson (u32)tv.tv_nsec; 3232da5c78c8SWilliam Allen Simpson 3233da5c78c8SWilliam Allen Simpson tcp_secret_secondary->expires = jiffy 3234da5c78c8SWilliam Allen Simpson + TCP_SECRET_1MSL 3235da5c78c8SWilliam Allen Simpson + (0x0f & tcp_cookie_work(bakery, 0)); 3236da5c78c8SWilliam Allen Simpson } else { 3237da5c78c8SWilliam Allen Simpson tcp_secret_secondary->expires = jiffy 3238da5c78c8SWilliam Allen Simpson + TCP_SECRET_LIFE 3239da5c78c8SWilliam Allen Simpson + (0xff & tcp_cookie_work(bakery, 1)); 3240da5c78c8SWilliam Allen Simpson tcp_secret_primary->expires = jiffy 3241da5c78c8SWilliam Allen Simpson + TCP_SECRET_2MSL 3242da5c78c8SWilliam Allen Simpson + (0x1f & tcp_cookie_work(bakery, 2)); 3243da5c78c8SWilliam Allen Simpson } 3244da5c78c8SWilliam Allen Simpson memcpy(&tcp_secret_secondary->secrets[0], 3245da5c78c8SWilliam Allen Simpson bakery, COOKIE_WORKSPACE_WORDS); 3246da5c78c8SWilliam Allen Simpson 3247da5c78c8SWilliam Allen Simpson rcu_assign_pointer(tcp_secret_generating, 3248da5c78c8SWilliam Allen Simpson tcp_secret_secondary); 3249da5c78c8SWilliam Allen Simpson rcu_assign_pointer(tcp_secret_retiring, 3250da5c78c8SWilliam Allen Simpson tcp_secret_primary); 3251da5c78c8SWilliam Allen Simpson /* 3252da5c78c8SWilliam Allen Simpson * Neither call_rcu() nor synchronize_rcu() needed. 3253da5c78c8SWilliam Allen Simpson * Retiring data is not freed. It is replaced after 3254da5c78c8SWilliam Allen Simpson * further (locked) pointer updates, and a quiet time 3255da5c78c8SWilliam Allen Simpson * (minimum 1MSL, maximum LIFE - 2MSL). 3256da5c78c8SWilliam Allen Simpson */ 3257da5c78c8SWilliam Allen Simpson } 3258da5c78c8SWilliam Allen Simpson spin_unlock_bh(&tcp_secret_locker); 3259da5c78c8SWilliam Allen Simpson } else { 3260da5c78c8SWilliam Allen Simpson rcu_read_lock_bh(); 3261da5c78c8SWilliam Allen Simpson memcpy(bakery, 3262da5c78c8SWilliam Allen Simpson &rcu_dereference(tcp_secret_generating)->secrets[0], 3263da5c78c8SWilliam Allen Simpson COOKIE_WORKSPACE_WORDS); 3264da5c78c8SWilliam Allen Simpson rcu_read_unlock_bh(); 3265da5c78c8SWilliam Allen Simpson } 3266da5c78c8SWilliam Allen Simpson return 0; 3267da5c78c8SWilliam Allen Simpson } 3268da5c78c8SWilliam Allen Simpson EXPORT_SYMBOL(tcp_cookie_generator); 3269da5c78c8SWilliam Allen Simpson 32704ac02babSAndi Kleen void tcp_done(struct sock *sk) 32714ac02babSAndi Kleen { 32724ac02babSAndi Kleen if (sk->sk_state == TCP_SYN_SENT || sk->sk_state == TCP_SYN_RECV) 327363231bddSPavel Emelyanov TCP_INC_STATS_BH(sock_net(sk), TCP_MIB_ATTEMPTFAILS); 32744ac02babSAndi Kleen 32754ac02babSAndi Kleen tcp_set_state(sk, TCP_CLOSE); 32764ac02babSAndi Kleen tcp_clear_xmit_timers(sk); 32774ac02babSAndi Kleen 32784ac02babSAndi Kleen sk->sk_shutdown = SHUTDOWN_MASK; 32794ac02babSAndi Kleen 32804ac02babSAndi Kleen if (!sock_flag(sk, SOCK_DEAD)) 32814ac02babSAndi Kleen sk->sk_state_change(sk); 32824ac02babSAndi Kleen else 32834ac02babSAndi Kleen inet_csk_destroy_sock(sk); 32844ac02babSAndi Kleen } 32854ac02babSAndi Kleen EXPORT_SYMBOL_GPL(tcp_done); 32864ac02babSAndi Kleen 32875f8ef48dSStephen Hemminger extern struct tcp_congestion_ops tcp_reno; 32881da177e4SLinus Torvalds 32891da177e4SLinus Torvalds static __initdata unsigned long thash_entries; 32901da177e4SLinus Torvalds static int __init set_thash_entries(char *str) 32911da177e4SLinus Torvalds { 32921da177e4SLinus Torvalds if (!str) 32931da177e4SLinus Torvalds return 0; 32941da177e4SLinus Torvalds thash_entries = simple_strtoul(str, &str, 0); 32951da177e4SLinus Torvalds return 1; 32961da177e4SLinus Torvalds } 32971da177e4SLinus Torvalds __setup("thash_entries=", set_thash_entries); 32981da177e4SLinus Torvalds 32994acb4190SGlauber Costa void tcp_init_mem(struct net *net) 33004acb4190SGlauber Costa { 33014acb4190SGlauber Costa unsigned long limit = nr_free_buffer_pages() / 8; 33024acb4190SGlauber Costa limit = max(limit, 128UL); 33034acb4190SGlauber Costa net->ipv4.sysctl_tcp_mem[0] = limit / 4 * 3; 33044acb4190SGlauber Costa net->ipv4.sysctl_tcp_mem[1] = limit; 33054acb4190SGlauber Costa net->ipv4.sysctl_tcp_mem[2] = net->ipv4.sysctl_tcp_mem[0] * 2; 33064acb4190SGlauber Costa } 33074acb4190SGlauber Costa 33081da177e4SLinus Torvalds void __init tcp_init(void) 33091da177e4SLinus Torvalds { 33101da177e4SLinus Torvalds struct sk_buff *skb = NULL; 3311f03d78dbSEric Dumazet unsigned long limit; 3312074b8517SDimitri Sivanich int max_share, cnt; 3313074b8517SDimitri Sivanich unsigned int i; 3314da5c78c8SWilliam Allen Simpson unsigned long jiffy = jiffies; 33151da177e4SLinus Torvalds 33161f9e636eSPavel Emelyanov BUILD_BUG_ON(sizeof(struct tcp_skb_cb) > sizeof(skb->cb)); 33171da177e4SLinus Torvalds 33181748376bSEric Dumazet percpu_counter_init(&tcp_sockets_allocated, 0); 3319dd24c001SEric Dumazet percpu_counter_init(&tcp_orphan_count, 0); 33206e04e021SArnaldo Carvalho de Melo tcp_hashinfo.bind_bucket_cachep = 33216e04e021SArnaldo Carvalho de Melo kmem_cache_create("tcp_bind_bucket", 33226e04e021SArnaldo Carvalho de Melo sizeof(struct inet_bind_bucket), 0, 332320c2df83SPaul Mundt SLAB_HWCACHE_ALIGN|SLAB_PANIC, NULL); 33241da177e4SLinus Torvalds 33251da177e4SLinus Torvalds /* Size and allocate the main established and bind bucket 33261da177e4SLinus Torvalds * hash tables. 33271da177e4SLinus Torvalds * 33281da177e4SLinus Torvalds * The methodology is similar to that of the buffer cache. 33291da177e4SLinus Torvalds */ 33306e04e021SArnaldo Carvalho de Melo tcp_hashinfo.ehash = 33311da177e4SLinus Torvalds alloc_large_system_hash("TCP established", 33320f7ff927SArnaldo Carvalho de Melo sizeof(struct inet_ehash_bucket), 33331da177e4SLinus Torvalds thash_entries, 33344481374cSJan Beulich (totalram_pages >= 128 * 1024) ? 333518955cfcSMike Stroyan 13 : 15, 33369e950efaSJohn Heffner 0, 33371da177e4SLinus Torvalds NULL, 3338f373b53bSEric Dumazet &tcp_hashinfo.ehash_mask, 33390ccfe618SJean Delvare thash_entries ? 0 : 512 * 1024); 3340f373b53bSEric Dumazet for (i = 0; i <= tcp_hashinfo.ehash_mask; i++) { 33413ab5aee7SEric Dumazet INIT_HLIST_NULLS_HEAD(&tcp_hashinfo.ehash[i].chain, i); 33423ab5aee7SEric Dumazet INIT_HLIST_NULLS_HEAD(&tcp_hashinfo.ehash[i].twchain, i); 33431da177e4SLinus Torvalds } 3344230140cfSEric Dumazet if (inet_ehash_locks_alloc(&tcp_hashinfo)) 3345230140cfSEric Dumazet panic("TCP: failed to alloc ehash_locks"); 33466e04e021SArnaldo Carvalho de Melo tcp_hashinfo.bhash = 33471da177e4SLinus Torvalds alloc_large_system_hash("TCP bind", 33480f7ff927SArnaldo Carvalho de Melo sizeof(struct inet_bind_hashbucket), 3349f373b53bSEric Dumazet tcp_hashinfo.ehash_mask + 1, 33504481374cSJan Beulich (totalram_pages >= 128 * 1024) ? 335118955cfcSMike Stroyan 13 : 15, 33529e950efaSJohn Heffner 0, 33536e04e021SArnaldo Carvalho de Melo &tcp_hashinfo.bhash_size, 33541da177e4SLinus Torvalds NULL, 33551da177e4SLinus Torvalds 64 * 1024); 3356074b8517SDimitri Sivanich tcp_hashinfo.bhash_size = 1U << tcp_hashinfo.bhash_size; 33576e04e021SArnaldo Carvalho de Melo for (i = 0; i < tcp_hashinfo.bhash_size; i++) { 33586e04e021SArnaldo Carvalho de Melo spin_lock_init(&tcp_hashinfo.bhash[i].lock); 33596e04e021SArnaldo Carvalho de Melo INIT_HLIST_HEAD(&tcp_hashinfo.bhash[i].chain); 33601da177e4SLinus Torvalds } 33611da177e4SLinus Torvalds 3362c5ed63d6SEric Dumazet 3363c5ed63d6SEric Dumazet cnt = tcp_hashinfo.ehash_mask + 1; 3364c5ed63d6SEric Dumazet 3365c5ed63d6SEric Dumazet tcp_death_row.sysctl_max_tw_buckets = cnt / 2; 3366c5ed63d6SEric Dumazet sysctl_tcp_max_orphans = cnt / 2; 3367c5ed63d6SEric Dumazet sysctl_max_syn_backlog = max(128, cnt / 256); 33681da177e4SLinus Torvalds 33694acb4190SGlauber Costa tcp_init_mem(&init_net); 3370c43b874dSJason Wang /* Set per-socket limits to no more than 1/128 the pressure threshold */ 33715fb84b14SEric Dumazet limit = nr_free_buffer_pages() << (PAGE_SHIFT - 7); 33727b4f4b5eSJohn Heffner max_share = min(4UL*1024*1024, limit); 33737b4f4b5eSJohn Heffner 33743ab224beSHideo Aoki sysctl_tcp_wmem[0] = SK_MEM_QUANTUM; 33757b4f4b5eSJohn Heffner sysctl_tcp_wmem[1] = 16*1024; 33767b4f4b5eSJohn Heffner sysctl_tcp_wmem[2] = max(64*1024, max_share); 33777b4f4b5eSJohn Heffner 33783ab224beSHideo Aoki sysctl_tcp_rmem[0] = SK_MEM_QUANTUM; 33797b4f4b5eSJohn Heffner sysctl_tcp_rmem[1] = 87380; 33807b4f4b5eSJohn Heffner sysctl_tcp_rmem[2] = max(87380, max_share); 33811da177e4SLinus Torvalds 3382afd46503SJoe Perches pr_info("Hash tables configured (established %u bind %u)\n", 3383f373b53bSEric Dumazet tcp_hashinfo.ehash_mask + 1, tcp_hashinfo.bhash_size); 3384317a76f9SStephen Hemminger 3385317a76f9SStephen Hemminger tcp_register_congestion_control(&tcp_reno); 3386da5c78c8SWilliam Allen Simpson 3387da5c78c8SWilliam Allen Simpson memset(&tcp_secret_one.secrets[0], 0, sizeof(tcp_secret_one.secrets)); 3388da5c78c8SWilliam Allen Simpson memset(&tcp_secret_two.secrets[0], 0, sizeof(tcp_secret_two.secrets)); 3389da5c78c8SWilliam Allen Simpson tcp_secret_one.expires = jiffy; /* past due */ 3390da5c78c8SWilliam Allen Simpson tcp_secret_two.expires = jiffy; /* past due */ 3391da5c78c8SWilliam Allen Simpson tcp_secret_generating = &tcp_secret_one; 3392da5c78c8SWilliam Allen Simpson tcp_secret_primary = &tcp_secret_one; 3393da5c78c8SWilliam Allen Simpson tcp_secret_retiring = &tcp_secret_two; 3394da5c78c8SWilliam Allen Simpson tcp_secret_secondary = &tcp_secret_two; 33951da177e4SLinus Torvalds } 3396