11da177e4SLinus Torvalds /* 21da177e4SLinus Torvalds * INET An implementation of the TCP/IP protocol suite for the LINUX 31da177e4SLinus Torvalds * operating system. INET is implemented using the BSD Socket 41da177e4SLinus Torvalds * interface as the means of communication with the user level. 51da177e4SLinus Torvalds * 61da177e4SLinus Torvalds * Implementation of the Transmission Control Protocol(TCP). 71da177e4SLinus Torvalds * 802c30a84SJesper Juhl * Authors: Ross Biro 91da177e4SLinus Torvalds * Fred N. van Kempen, <waltje@uWalt.NL.Mugnet.ORG> 101da177e4SLinus Torvalds * Mark Evans, <evansmp@uhura.aston.ac.uk> 111da177e4SLinus Torvalds * Corey Minyard <wf-rch!minyard@relay.EU.net> 121da177e4SLinus Torvalds * Florian La Roche, <flla@stud.uni-sb.de> 131da177e4SLinus Torvalds * Charles Hedrick, <hedrick@klinzhai.rutgers.edu> 141da177e4SLinus Torvalds * Linus Torvalds, <torvalds@cs.helsinki.fi> 151da177e4SLinus Torvalds * Alan Cox, <gw4pts@gw4pts.ampr.org> 161da177e4SLinus Torvalds * Matthew Dillon, <dillon@apollo.west.oic.com> 171da177e4SLinus Torvalds * Arnt Gulbrandsen, <agulbra@nvg.unit.no> 181da177e4SLinus Torvalds * Jorge Cwik, <jorge@laser.satlink.net> 191da177e4SLinus Torvalds * 201da177e4SLinus Torvalds * Fixes: 211da177e4SLinus Torvalds * Alan Cox : Numerous verify_area() calls 221da177e4SLinus Torvalds * Alan Cox : Set the ACK bit on a reset 231da177e4SLinus Torvalds * Alan Cox : Stopped it crashing if it closed while 241da177e4SLinus Torvalds * sk->inuse=1 and was trying to connect 251da177e4SLinus Torvalds * (tcp_err()). 261da177e4SLinus Torvalds * Alan Cox : All icmp error handling was broken 271da177e4SLinus Torvalds * pointers passed where wrong and the 281da177e4SLinus Torvalds * socket was looked up backwards. Nobody 291da177e4SLinus Torvalds * tested any icmp error code obviously. 301da177e4SLinus Torvalds * Alan Cox : tcp_err() now handled properly. It 311da177e4SLinus Torvalds * wakes people on errors. poll 321da177e4SLinus Torvalds * behaves and the icmp error race 331da177e4SLinus Torvalds * has gone by moving it into sock.c 341da177e4SLinus Torvalds * Alan Cox : tcp_send_reset() fixed to work for 351da177e4SLinus Torvalds * everything not just packets for 361da177e4SLinus Torvalds * unknown sockets. 371da177e4SLinus Torvalds * Alan Cox : tcp option processing. 381da177e4SLinus Torvalds * Alan Cox : Reset tweaked (still not 100%) [Had 391da177e4SLinus Torvalds * syn rule wrong] 401da177e4SLinus Torvalds * Herp Rosmanith : More reset fixes 411da177e4SLinus Torvalds * Alan Cox : No longer acks invalid rst frames. 421da177e4SLinus Torvalds * Acking any kind of RST is right out. 431da177e4SLinus Torvalds * Alan Cox : Sets an ignore me flag on an rst 441da177e4SLinus Torvalds * receive otherwise odd bits of prattle 451da177e4SLinus Torvalds * escape still 461da177e4SLinus Torvalds * Alan Cox : Fixed another acking RST frame bug. 471da177e4SLinus Torvalds * Should stop LAN workplace lockups. 481da177e4SLinus Torvalds * Alan Cox : Some tidyups using the new skb list 491da177e4SLinus Torvalds * facilities 501da177e4SLinus Torvalds * Alan Cox : sk->keepopen now seems to work 511da177e4SLinus Torvalds * Alan Cox : Pulls options out correctly on accepts 521da177e4SLinus Torvalds * Alan Cox : Fixed assorted sk->rqueue->next errors 531da177e4SLinus Torvalds * Alan Cox : PSH doesn't end a TCP read. Switched a 541da177e4SLinus Torvalds * bit to skb ops. 551da177e4SLinus Torvalds * Alan Cox : Tidied tcp_data to avoid a potential 561da177e4SLinus Torvalds * nasty. 571da177e4SLinus Torvalds * Alan Cox : Added some better commenting, as the 581da177e4SLinus Torvalds * tcp is hard to follow 591da177e4SLinus Torvalds * Alan Cox : Removed incorrect check for 20 * psh 601da177e4SLinus Torvalds * Michael O'Reilly : ack < copied bug fix. 611da177e4SLinus Torvalds * Johannes Stille : Misc tcp fixes (not all in yet). 621da177e4SLinus Torvalds * Alan Cox : FIN with no memory -> CRASH 631da177e4SLinus Torvalds * Alan Cox : Added socket option proto entries. 641da177e4SLinus Torvalds * Also added awareness of them to accept. 651da177e4SLinus Torvalds * Alan Cox : Added TCP options (SOL_TCP) 661da177e4SLinus Torvalds * Alan Cox : Switched wakeup calls to callbacks, 671da177e4SLinus Torvalds * so the kernel can layer network 681da177e4SLinus Torvalds * sockets. 691da177e4SLinus Torvalds * Alan Cox : Use ip_tos/ip_ttl settings. 701da177e4SLinus Torvalds * Alan Cox : Handle FIN (more) properly (we hope). 711da177e4SLinus Torvalds * Alan Cox : RST frames sent on unsynchronised 721da177e4SLinus Torvalds * state ack error. 731da177e4SLinus Torvalds * Alan Cox : Put in missing check for SYN bit. 741da177e4SLinus Torvalds * Alan Cox : Added tcp_select_window() aka NET2E 751da177e4SLinus Torvalds * window non shrink trick. 761da177e4SLinus Torvalds * Alan Cox : Added a couple of small NET2E timer 771da177e4SLinus Torvalds * fixes 781da177e4SLinus Torvalds * Charles Hedrick : TCP fixes 791da177e4SLinus Torvalds * Toomas Tamm : TCP window fixes 801da177e4SLinus Torvalds * Alan Cox : Small URG fix to rlogin ^C ack fight 811da177e4SLinus Torvalds * Charles Hedrick : Rewrote most of it to actually work 821da177e4SLinus Torvalds * Linus : Rewrote tcp_read() and URG handling 831da177e4SLinus Torvalds * completely 841da177e4SLinus Torvalds * Gerhard Koerting: Fixed some missing timer handling 851da177e4SLinus Torvalds * Matthew Dillon : Reworked TCP machine states as per RFC 861da177e4SLinus Torvalds * Gerhard Koerting: PC/TCP workarounds 871da177e4SLinus Torvalds * Adam Caldwell : Assorted timer/timing errors 881da177e4SLinus Torvalds * Matthew Dillon : Fixed another RST bug 891da177e4SLinus Torvalds * Alan Cox : Move to kernel side addressing changes. 901da177e4SLinus Torvalds * Alan Cox : Beginning work on TCP fastpathing 911da177e4SLinus Torvalds * (not yet usable) 921da177e4SLinus Torvalds * Arnt Gulbrandsen: Turbocharged tcp_check() routine. 931da177e4SLinus Torvalds * Alan Cox : TCP fast path debugging 941da177e4SLinus Torvalds * Alan Cox : Window clamping 951da177e4SLinus Torvalds * Michael Riepe : Bug in tcp_check() 961da177e4SLinus Torvalds * Matt Dillon : More TCP improvements and RST bug fixes 971da177e4SLinus Torvalds * Matt Dillon : Yet more small nasties remove from the 981da177e4SLinus Torvalds * TCP code (Be very nice to this man if 991da177e4SLinus Torvalds * tcp finally works 100%) 8) 1001da177e4SLinus Torvalds * Alan Cox : BSD accept semantics. 1011da177e4SLinus Torvalds * Alan Cox : Reset on closedown bug. 1021da177e4SLinus Torvalds * Peter De Schrijver : ENOTCONN check missing in tcp_sendto(). 1031da177e4SLinus Torvalds * Michael Pall : Handle poll() after URG properly in 1041da177e4SLinus Torvalds * all cases. 1051da177e4SLinus Torvalds * Michael Pall : Undo the last fix in tcp_read_urg() 1061da177e4SLinus Torvalds * (multi URG PUSH broke rlogin). 1071da177e4SLinus Torvalds * Michael Pall : Fix the multi URG PUSH problem in 1081da177e4SLinus Torvalds * tcp_readable(), poll() after URG 1091da177e4SLinus Torvalds * works now. 1101da177e4SLinus Torvalds * Michael Pall : recv(...,MSG_OOB) never blocks in the 1111da177e4SLinus Torvalds * BSD api. 1121da177e4SLinus Torvalds * Alan Cox : Changed the semantics of sk->socket to 1131da177e4SLinus Torvalds * fix a race and a signal problem with 1141da177e4SLinus Torvalds * accept() and async I/O. 1151da177e4SLinus Torvalds * Alan Cox : Relaxed the rules on tcp_sendto(). 1161da177e4SLinus Torvalds * Yury Shevchuk : Really fixed accept() blocking problem. 1171da177e4SLinus Torvalds * Craig I. Hagan : Allow for BSD compatible TIME_WAIT for 1181da177e4SLinus Torvalds * clients/servers which listen in on 1191da177e4SLinus Torvalds * fixed ports. 1201da177e4SLinus Torvalds * Alan Cox : Cleaned the above up and shrank it to 1211da177e4SLinus Torvalds * a sensible code size. 1221da177e4SLinus Torvalds * Alan Cox : Self connect lockup fix. 1231da177e4SLinus Torvalds * Alan Cox : No connect to multicast. 1241da177e4SLinus Torvalds * Ross Biro : Close unaccepted children on master 1251da177e4SLinus Torvalds * socket close. 1261da177e4SLinus Torvalds * Alan Cox : Reset tracing code. 1271da177e4SLinus Torvalds * Alan Cox : Spurious resets on shutdown. 1281da177e4SLinus Torvalds * Alan Cox : Giant 15 minute/60 second timer error 1291da177e4SLinus Torvalds * Alan Cox : Small whoops in polling before an 1301da177e4SLinus Torvalds * accept. 1311da177e4SLinus Torvalds * Alan Cox : Kept the state trace facility since 1321da177e4SLinus Torvalds * it's handy for debugging. 1331da177e4SLinus Torvalds * Alan Cox : More reset handler fixes. 1341da177e4SLinus Torvalds * Alan Cox : Started rewriting the code based on 1351da177e4SLinus Torvalds * the RFC's for other useful protocol 1361da177e4SLinus Torvalds * references see: Comer, KA9Q NOS, and 1371da177e4SLinus Torvalds * for a reference on the difference 1381da177e4SLinus Torvalds * between specifications and how BSD 1391da177e4SLinus Torvalds * works see the 4.4lite source. 1401da177e4SLinus Torvalds * A.N.Kuznetsov : Don't time wait on completion of tidy 1411da177e4SLinus Torvalds * close. 1421da177e4SLinus Torvalds * Linus Torvalds : Fin/Shutdown & copied_seq changes. 1431da177e4SLinus Torvalds * Linus Torvalds : Fixed BSD port reuse to work first syn 1441da177e4SLinus Torvalds * Alan Cox : Reimplemented timers as per the RFC 1451da177e4SLinus Torvalds * and using multiple timers for sanity. 1461da177e4SLinus Torvalds * Alan Cox : Small bug fixes, and a lot of new 1471da177e4SLinus Torvalds * comments. 1481da177e4SLinus Torvalds * Alan Cox : Fixed dual reader crash by locking 1491da177e4SLinus Torvalds * the buffers (much like datagram.c) 1501da177e4SLinus Torvalds * Alan Cox : Fixed stuck sockets in probe. A probe 1511da177e4SLinus Torvalds * now gets fed up of retrying without 1521da177e4SLinus Torvalds * (even a no space) answer. 1531da177e4SLinus Torvalds * Alan Cox : Extracted closing code better 1541da177e4SLinus Torvalds * Alan Cox : Fixed the closing state machine to 1551da177e4SLinus Torvalds * resemble the RFC. 1561da177e4SLinus Torvalds * Alan Cox : More 'per spec' fixes. 1571da177e4SLinus Torvalds * Jorge Cwik : Even faster checksumming. 1581da177e4SLinus Torvalds * Alan Cox : tcp_data() doesn't ack illegal PSH 1591da177e4SLinus Torvalds * only frames. At least one pc tcp stack 1601da177e4SLinus Torvalds * generates them. 1611da177e4SLinus Torvalds * Alan Cox : Cache last socket. 1621da177e4SLinus Torvalds * Alan Cox : Per route irtt. 1631da177e4SLinus Torvalds * Matt Day : poll()->select() match BSD precisely on error 1641da177e4SLinus Torvalds * Alan Cox : New buffers 1651da177e4SLinus Torvalds * Marc Tamsky : Various sk->prot->retransmits and 1661da177e4SLinus Torvalds * sk->retransmits misupdating fixed. 1671da177e4SLinus Torvalds * Fixed tcp_write_timeout: stuck close, 1681da177e4SLinus Torvalds * and TCP syn retries gets used now. 1691da177e4SLinus Torvalds * Mark Yarvis : In tcp_read_wakeup(), don't send an 1701da177e4SLinus Torvalds * ack if state is TCP_CLOSED. 1711da177e4SLinus Torvalds * Alan Cox : Look up device on a retransmit - routes may 1721da177e4SLinus Torvalds * change. Doesn't yet cope with MSS shrink right 1731da177e4SLinus Torvalds * but it's a start! 1741da177e4SLinus Torvalds * Marc Tamsky : Closing in closing fixes. 1751da177e4SLinus Torvalds * Mike Shaver : RFC1122 verifications. 1761da177e4SLinus Torvalds * Alan Cox : rcv_saddr errors. 1771da177e4SLinus Torvalds * Alan Cox : Block double connect(). 1781da177e4SLinus Torvalds * Alan Cox : Small hooks for enSKIP. 1791da177e4SLinus Torvalds * Alexey Kuznetsov: Path MTU discovery. 1801da177e4SLinus Torvalds * Alan Cox : Support soft errors. 1811da177e4SLinus Torvalds * Alan Cox : Fix MTU discovery pathological case 1821da177e4SLinus Torvalds * when the remote claims no mtu! 1831da177e4SLinus Torvalds * Marc Tamsky : TCP_CLOSE fix. 1841da177e4SLinus Torvalds * Colin (G3TNE) : Send a reset on syn ack replies in 1851da177e4SLinus Torvalds * window but wrong (fixes NT lpd problems) 1861da177e4SLinus Torvalds * Pedro Roque : Better TCP window handling, delayed ack. 1871da177e4SLinus Torvalds * Joerg Reuter : No modification of locked buffers in 1881da177e4SLinus Torvalds * tcp_do_retransmit() 1891da177e4SLinus Torvalds * Eric Schenk : Changed receiver side silly window 1901da177e4SLinus Torvalds * avoidance algorithm to BSD style 1911da177e4SLinus Torvalds * algorithm. This doubles throughput 1921da177e4SLinus Torvalds * against machines running Solaris, 1931da177e4SLinus Torvalds * and seems to result in general 1941da177e4SLinus Torvalds * improvement. 1951da177e4SLinus Torvalds * Stefan Magdalinski : adjusted tcp_readable() to fix FIONREAD 1961da177e4SLinus Torvalds * Willy Konynenberg : Transparent proxying support. 1971da177e4SLinus Torvalds * Mike McLagan : Routing by source 1981da177e4SLinus Torvalds * Keith Owens : Do proper merging with partial SKB's in 1991da177e4SLinus Torvalds * tcp_do_sendmsg to avoid burstiness. 2001da177e4SLinus Torvalds * Eric Schenk : Fix fast close down bug with 2011da177e4SLinus Torvalds * shutdown() followed by close(). 2021da177e4SLinus Torvalds * Andi Kleen : Make poll agree with SIGIO 2031da177e4SLinus Torvalds * Salvatore Sanfilippo : Support SO_LINGER with linger == 1 and 2041da177e4SLinus Torvalds * lingertime == 0 (RFC 793 ABORT Call) 2051da177e4SLinus Torvalds * Hirokazu Takahashi : Use copy_from_user() instead of 2061da177e4SLinus Torvalds * csum_and_copy_from_user() if possible. 2071da177e4SLinus Torvalds * 2081da177e4SLinus Torvalds * This program is free software; you can redistribute it and/or 2091da177e4SLinus Torvalds * modify it under the terms of the GNU General Public License 2101da177e4SLinus Torvalds * as published by the Free Software Foundation; either version 2111da177e4SLinus Torvalds * 2 of the License, or(at your option) any later version. 2121da177e4SLinus Torvalds * 2131da177e4SLinus Torvalds * Description of States: 2141da177e4SLinus Torvalds * 2151da177e4SLinus Torvalds * TCP_SYN_SENT sent a connection request, waiting for ack 2161da177e4SLinus Torvalds * 2171da177e4SLinus Torvalds * TCP_SYN_RECV received a connection request, sent ack, 2181da177e4SLinus Torvalds * waiting for final ack in three-way handshake. 2191da177e4SLinus Torvalds * 2201da177e4SLinus Torvalds * TCP_ESTABLISHED connection established 2211da177e4SLinus Torvalds * 2221da177e4SLinus Torvalds * TCP_FIN_WAIT1 our side has shutdown, waiting to complete 2231da177e4SLinus Torvalds * transmission of remaining buffered data 2241da177e4SLinus Torvalds * 2251da177e4SLinus Torvalds * TCP_FIN_WAIT2 all buffered data sent, waiting for remote 2261da177e4SLinus Torvalds * to shutdown 2271da177e4SLinus Torvalds * 2281da177e4SLinus Torvalds * TCP_CLOSING both sides have shutdown but we still have 2291da177e4SLinus Torvalds * data we have to finish sending 2301da177e4SLinus Torvalds * 2311da177e4SLinus Torvalds * TCP_TIME_WAIT timeout to catch resent junk before entering 2321da177e4SLinus Torvalds * closed, can only be entered from FIN_WAIT2 2331da177e4SLinus Torvalds * or CLOSING. Required because the other end 2341da177e4SLinus Torvalds * may not have gotten our last ACK causing it 2351da177e4SLinus Torvalds * to retransmit the data packet (which we ignore) 2361da177e4SLinus Torvalds * 2371da177e4SLinus Torvalds * TCP_CLOSE_WAIT remote side has shutdown and is waiting for 2381da177e4SLinus Torvalds * us to finish writing our data and to shutdown 2391da177e4SLinus Torvalds * (we have to close() to move on to LAST_ACK) 2401da177e4SLinus Torvalds * 2411da177e4SLinus Torvalds * TCP_LAST_ACK out side has shutdown after remote has 2421da177e4SLinus Torvalds * shutdown. There may still be data in our 2431da177e4SLinus Torvalds * buffer that we have to finish sending 2441da177e4SLinus Torvalds * 2451da177e4SLinus Torvalds * TCP_CLOSE socket is finished 2461da177e4SLinus Torvalds */ 2471da177e4SLinus Torvalds 248172589ccSIlpo Järvinen #include <linux/kernel.h> 2491da177e4SLinus Torvalds #include <linux/module.h> 2501da177e4SLinus Torvalds #include <linux/types.h> 2511da177e4SLinus Torvalds #include <linux/fcntl.h> 2521da177e4SLinus Torvalds #include <linux/poll.h> 2531da177e4SLinus Torvalds #include <linux/init.h> 2541da177e4SLinus Torvalds #include <linux/fs.h> 2559c55e01cSJens Axboe #include <linux/skbuff.h> 25681b23b4aSAndrew Morton #include <linux/scatterlist.h> 2579c55e01cSJens Axboe #include <linux/splice.h> 2589c55e01cSJens Axboe #include <linux/net.h> 2599c55e01cSJens Axboe #include <linux/socket.h> 2601da177e4SLinus Torvalds #include <linux/random.h> 2611da177e4SLinus Torvalds #include <linux/bootmem.h> 26257413ebcSMiquel van Smoorenburg #include <linux/highmem.h> 26357413ebcSMiquel van Smoorenburg #include <linux/swap.h> 264b8059eadSDavid S. Miller #include <linux/cache.h> 265f4c50d99SHerbert Xu #include <linux/err.h> 266cfb6eeb4SYOSHIFUJI Hideaki #include <linux/crypto.h> 267da5c78c8SWilliam Allen Simpson #include <linux/time.h> 2685a0e3ad6STejun Heo #include <linux/slab.h> 2691da177e4SLinus Torvalds 2701da177e4SLinus Torvalds #include <net/icmp.h> 2711da177e4SLinus Torvalds #include <net/tcp.h> 2721da177e4SLinus Torvalds #include <net/xfrm.h> 2731da177e4SLinus Torvalds #include <net/ip.h> 2741a2449a8SChris Leech #include <net/netdma.h> 2759c55e01cSJens Axboe #include <net/sock.h> 2761da177e4SLinus Torvalds 2771da177e4SLinus Torvalds #include <asm/uaccess.h> 2781da177e4SLinus Torvalds #include <asm/ioctls.h> 2791da177e4SLinus Torvalds 280ab32ea5dSBrian Haley int sysctl_tcp_fin_timeout __read_mostly = TCP_FIN_TIMEOUT; 2811da177e4SLinus Torvalds 282dd24c001SEric Dumazet struct percpu_counter tcp_orphan_count; 2830a5578cfSArnaldo Carvalho de Melo EXPORT_SYMBOL_GPL(tcp_orphan_count); 2840a5578cfSArnaldo Carvalho de Melo 2858d987e5cSEric Dumazet long sysctl_tcp_mem[3] __read_mostly; 286b8059eadSDavid S. Miller int sysctl_tcp_wmem[3] __read_mostly; 287b8059eadSDavid S. Miller int sysctl_tcp_rmem[3] __read_mostly; 2881da177e4SLinus Torvalds 2891da177e4SLinus Torvalds EXPORT_SYMBOL(sysctl_tcp_mem); 2901da177e4SLinus Torvalds EXPORT_SYMBOL(sysctl_tcp_rmem); 2911da177e4SLinus Torvalds EXPORT_SYMBOL(sysctl_tcp_wmem); 2921da177e4SLinus Torvalds 2938d987e5cSEric Dumazet atomic_long_t tcp_memory_allocated; /* Current allocated memory. */ 2941da177e4SLinus Torvalds EXPORT_SYMBOL(tcp_memory_allocated); 2951748376bSEric Dumazet 2961748376bSEric Dumazet /* 2971748376bSEric Dumazet * Current number of TCP sockets. 2981748376bSEric Dumazet */ 2991748376bSEric Dumazet struct percpu_counter tcp_sockets_allocated; 3001da177e4SLinus Torvalds EXPORT_SYMBOL(tcp_sockets_allocated); 3011da177e4SLinus Torvalds 3021da177e4SLinus Torvalds /* 3039c55e01cSJens Axboe * TCP splice context 3049c55e01cSJens Axboe */ 3059c55e01cSJens Axboe struct tcp_splice_state { 3069c55e01cSJens Axboe struct pipe_inode_info *pipe; 3079c55e01cSJens Axboe size_t len; 3089c55e01cSJens Axboe unsigned int flags; 3099c55e01cSJens Axboe }; 3109c55e01cSJens Axboe 3119c55e01cSJens Axboe /* 3121da177e4SLinus Torvalds * Pressure flag: try to collapse. 3131da177e4SLinus Torvalds * Technical note: it is used by multiple contexts non atomically. 3143ab224beSHideo Aoki * All the __sk_mem_schedule() is of this nature: accounting 3151da177e4SLinus Torvalds * is strict, actions are advisory and have some latency. 3161da177e4SLinus Torvalds */ 3174103f8cdSEric Dumazet int tcp_memory_pressure __read_mostly; 3181da177e4SLinus Torvalds EXPORT_SYMBOL(tcp_memory_pressure); 3191da177e4SLinus Torvalds 3205c52ba17SPavel Emelyanov void tcp_enter_memory_pressure(struct sock *sk) 3211da177e4SLinus Torvalds { 3221da177e4SLinus Torvalds if (!tcp_memory_pressure) { 3234e673444SPavel Emelyanov NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPMEMORYPRESSURES); 3241da177e4SLinus Torvalds tcp_memory_pressure = 1; 3251da177e4SLinus Torvalds } 3261da177e4SLinus Torvalds } 3271da177e4SLinus Torvalds EXPORT_SYMBOL(tcp_enter_memory_pressure); 3281da177e4SLinus Torvalds 329b103cf34SJulian Anastasov /* Convert seconds to retransmits based on initial and max timeout */ 330b103cf34SJulian Anastasov static u8 secs_to_retrans(int seconds, int timeout, int rto_max) 331b103cf34SJulian Anastasov { 332b103cf34SJulian Anastasov u8 res = 0; 333b103cf34SJulian Anastasov 334b103cf34SJulian Anastasov if (seconds > 0) { 335b103cf34SJulian Anastasov int period = timeout; 336b103cf34SJulian Anastasov 337b103cf34SJulian Anastasov res = 1; 338b103cf34SJulian Anastasov while (seconds > period && res < 255) { 339b103cf34SJulian Anastasov res++; 340b103cf34SJulian Anastasov timeout <<= 1; 341b103cf34SJulian Anastasov if (timeout > rto_max) 342b103cf34SJulian Anastasov timeout = rto_max; 343b103cf34SJulian Anastasov period += timeout; 344b103cf34SJulian Anastasov } 345b103cf34SJulian Anastasov } 346b103cf34SJulian Anastasov return res; 347b103cf34SJulian Anastasov } 348b103cf34SJulian Anastasov 349b103cf34SJulian Anastasov /* Convert retransmits to seconds based on initial and max timeout */ 350b103cf34SJulian Anastasov static int retrans_to_secs(u8 retrans, int timeout, int rto_max) 351b103cf34SJulian Anastasov { 352b103cf34SJulian Anastasov int period = 0; 353b103cf34SJulian Anastasov 354b103cf34SJulian Anastasov if (retrans > 0) { 355b103cf34SJulian Anastasov period = timeout; 356b103cf34SJulian Anastasov while (--retrans) { 357b103cf34SJulian Anastasov timeout <<= 1; 358b103cf34SJulian Anastasov if (timeout > rto_max) 359b103cf34SJulian Anastasov timeout = rto_max; 360b103cf34SJulian Anastasov period += timeout; 361b103cf34SJulian Anastasov } 362b103cf34SJulian Anastasov } 363b103cf34SJulian Anastasov return period; 364b103cf34SJulian Anastasov } 365b103cf34SJulian Anastasov 3661da177e4SLinus Torvalds /* 3671da177e4SLinus Torvalds * Wait for a TCP event. 3681da177e4SLinus Torvalds * 3691da177e4SLinus Torvalds * Note that we don't need to lock the socket, as the upper poll layers 3701da177e4SLinus Torvalds * take care of normal races (between the test and the event) and we don't 3711da177e4SLinus Torvalds * go look at any of the socket buffers directly. 3721da177e4SLinus Torvalds */ 3731da177e4SLinus Torvalds unsigned int tcp_poll(struct file *file, struct socket *sock, poll_table *wait) 3741da177e4SLinus Torvalds { 3751da177e4SLinus Torvalds unsigned int mask; 3761da177e4SLinus Torvalds struct sock *sk = sock->sk; 3771da177e4SLinus Torvalds struct tcp_sock *tp = tcp_sk(sk); 3781da177e4SLinus Torvalds 379aa395145SEric Dumazet sock_poll_wait(file, sk_sleep(sk), wait); 3801da177e4SLinus Torvalds if (sk->sk_state == TCP_LISTEN) 381dc40c7bcSArnaldo Carvalho de Melo return inet_csk_listen_poll(sk); 3821da177e4SLinus Torvalds 3831da177e4SLinus Torvalds /* Socket is not locked. We are protected from async events 38470efce27SWill Newton * by poll logic and correct handling of state changes 38570efce27SWill Newton * made by other threads is impossible in any case. 3861da177e4SLinus Torvalds */ 3871da177e4SLinus Torvalds 3881da177e4SLinus Torvalds mask = 0; 3891da177e4SLinus Torvalds 3901da177e4SLinus Torvalds /* 3911da177e4SLinus Torvalds * POLLHUP is certainly not done right. But poll() doesn't 3921da177e4SLinus Torvalds * have a notion of HUP in just one direction, and for a 3931da177e4SLinus Torvalds * socket the read side is more interesting. 3941da177e4SLinus Torvalds * 3951da177e4SLinus Torvalds * Some poll() documentation says that POLLHUP is incompatible 3961da177e4SLinus Torvalds * with the POLLOUT/POLLWR flags, so somebody should check this 3971da177e4SLinus Torvalds * all. But careful, it tends to be safer to return too many 3981da177e4SLinus Torvalds * bits than too few, and you can easily break real applications 3991da177e4SLinus Torvalds * if you don't tell them that something has hung up! 4001da177e4SLinus Torvalds * 4011da177e4SLinus Torvalds * Check-me. 4021da177e4SLinus Torvalds * 4031da177e4SLinus Torvalds * Check number 1. POLLHUP is _UNMASKABLE_ event (see UNIX98 and 4041da177e4SLinus Torvalds * our fs/select.c). It means that after we received EOF, 4051da177e4SLinus Torvalds * poll always returns immediately, making impossible poll() on write() 4061da177e4SLinus Torvalds * in state CLOSE_WAIT. One solution is evident --- to set POLLHUP 4071da177e4SLinus Torvalds * if and only if shutdown has been made in both directions. 4081da177e4SLinus Torvalds * Actually, it is interesting to look how Solaris and DUX 40970efce27SWill Newton * solve this dilemma. I would prefer, if POLLHUP were maskable, 4101da177e4SLinus Torvalds * then we could set it on SND_SHUTDOWN. BTW examples given 4111da177e4SLinus Torvalds * in Stevens' books assume exactly this behaviour, it explains 41270efce27SWill Newton * why POLLHUP is incompatible with POLLOUT. --ANK 4131da177e4SLinus Torvalds * 4141da177e4SLinus Torvalds * NOTE. Check for TCP_CLOSE is added. The goal is to prevent 4151da177e4SLinus Torvalds * blocking on fresh not-connected or disconnected socket. --ANK 4161da177e4SLinus Torvalds */ 4171da177e4SLinus Torvalds if (sk->sk_shutdown == SHUTDOWN_MASK || sk->sk_state == TCP_CLOSE) 4181da177e4SLinus Torvalds mask |= POLLHUP; 4191da177e4SLinus Torvalds if (sk->sk_shutdown & RCV_SHUTDOWN) 420f348d70aSDavide Libenzi mask |= POLLIN | POLLRDNORM | POLLRDHUP; 4211da177e4SLinus Torvalds 4221da177e4SLinus Torvalds /* Connected? */ 4231da177e4SLinus Torvalds if ((1 << sk->sk_state) & ~(TCPF_SYN_SENT | TCPF_SYN_RECV)) { 424c7004482SDavid S. Miller int target = sock_rcvlowat(sk, 0, INT_MAX); 425c7004482SDavid S. Miller 426c7004482SDavid S. Miller if (tp->urg_seq == tp->copied_seq && 427c7004482SDavid S. Miller !sock_flag(sk, SOCK_URGINLINE) && 428c7004482SDavid S. Miller tp->urg_data) 429b634f875SAlexandra Kossovsky target++; 430c7004482SDavid S. Miller 4311da177e4SLinus Torvalds /* Potential race condition. If read of tp below will 4321da177e4SLinus Torvalds * escape above sk->sk_state, we can be illegally awaken 4331da177e4SLinus Torvalds * in SYN_* states. */ 434c7004482SDavid S. Miller if (tp->rcv_nxt - tp->copied_seq >= target) 4351da177e4SLinus Torvalds mask |= POLLIN | POLLRDNORM; 4361da177e4SLinus Torvalds 4371da177e4SLinus Torvalds if (!(sk->sk_shutdown & SEND_SHUTDOWN)) { 4381da177e4SLinus Torvalds if (sk_stream_wspace(sk) >= sk_stream_min_wspace(sk)) { 4391da177e4SLinus Torvalds mask |= POLLOUT | POLLWRNORM; 4401da177e4SLinus Torvalds } else { /* send SIGIO later */ 4411da177e4SLinus Torvalds set_bit(SOCK_ASYNC_NOSPACE, 4421da177e4SLinus Torvalds &sk->sk_socket->flags); 4431da177e4SLinus Torvalds set_bit(SOCK_NOSPACE, &sk->sk_socket->flags); 4441da177e4SLinus Torvalds 4451da177e4SLinus Torvalds /* Race breaker. If space is freed after 4461da177e4SLinus Torvalds * wspace test but before the flags are set, 4471da177e4SLinus Torvalds * IO signal will be lost. 4481da177e4SLinus Torvalds */ 4491da177e4SLinus Torvalds if (sk_stream_wspace(sk) >= sk_stream_min_wspace(sk)) 4501da177e4SLinus Torvalds mask |= POLLOUT | POLLWRNORM; 4511da177e4SLinus Torvalds } 452d84ba638SKOSAKI Motohiro } else 453d84ba638SKOSAKI Motohiro mask |= POLLOUT | POLLWRNORM; 4541da177e4SLinus Torvalds 4551da177e4SLinus Torvalds if (tp->urg_data & TCP_URG_VALID) 4561da177e4SLinus Torvalds mask |= POLLPRI; 4571da177e4SLinus Torvalds } 458a4d25803STom Marshall /* This barrier is coupled with smp_wmb() in tcp_reset() */ 459a4d25803STom Marshall smp_rmb(); 460a4d25803STom Marshall if (sk->sk_err) 461a4d25803STom Marshall mask |= POLLERR; 462a4d25803STom Marshall 4631da177e4SLinus Torvalds return mask; 4641da177e4SLinus Torvalds } 4654bc2f18bSEric Dumazet EXPORT_SYMBOL(tcp_poll); 4661da177e4SLinus Torvalds 4671da177e4SLinus Torvalds int tcp_ioctl(struct sock *sk, int cmd, unsigned long arg) 4681da177e4SLinus Torvalds { 4691da177e4SLinus Torvalds struct tcp_sock *tp = tcp_sk(sk); 4701da177e4SLinus Torvalds int answ; 4711da177e4SLinus Torvalds 4721da177e4SLinus Torvalds switch (cmd) { 4731da177e4SLinus Torvalds case SIOCINQ: 4741da177e4SLinus Torvalds if (sk->sk_state == TCP_LISTEN) 4751da177e4SLinus Torvalds return -EINVAL; 4761da177e4SLinus Torvalds 4771da177e4SLinus Torvalds lock_sock(sk); 4781da177e4SLinus Torvalds if ((1 << sk->sk_state) & (TCPF_SYN_SENT | TCPF_SYN_RECV)) 4791da177e4SLinus Torvalds answ = 0; 4801da177e4SLinus Torvalds else if (sock_flag(sk, SOCK_URGINLINE) || 4811da177e4SLinus Torvalds !tp->urg_data || 4821da177e4SLinus Torvalds before(tp->urg_seq, tp->copied_seq) || 4831da177e4SLinus Torvalds !before(tp->urg_seq, tp->rcv_nxt)) { 48491521944SDavid S. Miller struct sk_buff *skb; 48591521944SDavid S. Miller 4861da177e4SLinus Torvalds answ = tp->rcv_nxt - tp->copied_seq; 4871da177e4SLinus Torvalds 4881da177e4SLinus Torvalds /* Subtract 1, if FIN is in queue. */ 48991521944SDavid S. Miller skb = skb_peek_tail(&sk->sk_receive_queue); 49091521944SDavid S. Miller if (answ && skb) 49191521944SDavid S. Miller answ -= tcp_hdr(skb)->fin; 4921da177e4SLinus Torvalds } else 4931da177e4SLinus Torvalds answ = tp->urg_seq - tp->copied_seq; 4941da177e4SLinus Torvalds release_sock(sk); 4951da177e4SLinus Torvalds break; 4961da177e4SLinus Torvalds case SIOCATMARK: 4971da177e4SLinus Torvalds answ = tp->urg_data && tp->urg_seq == tp->copied_seq; 4981da177e4SLinus Torvalds break; 4991da177e4SLinus Torvalds case SIOCOUTQ: 5001da177e4SLinus Torvalds if (sk->sk_state == TCP_LISTEN) 5011da177e4SLinus Torvalds return -EINVAL; 5021da177e4SLinus Torvalds 5031da177e4SLinus Torvalds if ((1 << sk->sk_state) & (TCPF_SYN_SENT | TCPF_SYN_RECV)) 5041da177e4SLinus Torvalds answ = 0; 5051da177e4SLinus Torvalds else 5061da177e4SLinus Torvalds answ = tp->write_seq - tp->snd_una; 5071da177e4SLinus Torvalds break; 5082f4e1b39SMario Schuknecht case SIOCOUTQNSD: 5092f4e1b39SMario Schuknecht if (sk->sk_state == TCP_LISTEN) 5102f4e1b39SMario Schuknecht return -EINVAL; 5112f4e1b39SMario Schuknecht 5122f4e1b39SMario Schuknecht if ((1 << sk->sk_state) & (TCPF_SYN_SENT | TCPF_SYN_RECV)) 5132f4e1b39SMario Schuknecht answ = 0; 5142f4e1b39SMario Schuknecht else 5152f4e1b39SMario Schuknecht answ = tp->write_seq - tp->snd_nxt; 5162f4e1b39SMario Schuknecht break; 5171da177e4SLinus Torvalds default: 5181da177e4SLinus Torvalds return -ENOIOCTLCMD; 5193ff50b79SStephen Hemminger } 5201da177e4SLinus Torvalds 5211da177e4SLinus Torvalds return put_user(answ, (int __user *)arg); 5221da177e4SLinus Torvalds } 5234bc2f18bSEric Dumazet EXPORT_SYMBOL(tcp_ioctl); 5241da177e4SLinus Torvalds 5251da177e4SLinus Torvalds static inline void tcp_mark_push(struct tcp_sock *tp, struct sk_buff *skb) 5261da177e4SLinus Torvalds { 5274de075e0SEric Dumazet TCP_SKB_CB(skb)->tcp_flags |= TCPHDR_PSH; 5281da177e4SLinus Torvalds tp->pushed_seq = tp->write_seq; 5291da177e4SLinus Torvalds } 5301da177e4SLinus Torvalds 5311da177e4SLinus Torvalds static inline int forced_push(struct tcp_sock *tp) 5321da177e4SLinus Torvalds { 5331da177e4SLinus Torvalds return after(tp->write_seq, tp->pushed_seq + (tp->max_window >> 1)); 5341da177e4SLinus Torvalds } 5351da177e4SLinus Torvalds 5369e412ba7SIlpo Järvinen static inline void skb_entail(struct sock *sk, struct sk_buff *skb) 5371da177e4SLinus Torvalds { 5389e412ba7SIlpo Järvinen struct tcp_sock *tp = tcp_sk(sk); 539352d4800SArnaldo Carvalho de Melo struct tcp_skb_cb *tcb = TCP_SKB_CB(skb); 540352d4800SArnaldo Carvalho de Melo 5411da177e4SLinus Torvalds skb->csum = 0; 542352d4800SArnaldo Carvalho de Melo tcb->seq = tcb->end_seq = tp->write_seq; 5434de075e0SEric Dumazet tcb->tcp_flags = TCPHDR_ACK; 544352d4800SArnaldo Carvalho de Melo tcb->sacked = 0; 5451da177e4SLinus Torvalds skb_header_release(skb); 546fe067e8aSDavid S. Miller tcp_add_write_queue_tail(sk, skb); 5473ab224beSHideo Aoki sk->sk_wmem_queued += skb->truesize; 5483ab224beSHideo Aoki sk_mem_charge(sk, skb->truesize); 54989ebd197SDavid S. Miller if (tp->nonagle & TCP_NAGLE_PUSH) 5501da177e4SLinus Torvalds tp->nonagle &= ~TCP_NAGLE_PUSH; 5511da177e4SLinus Torvalds } 5521da177e4SLinus Torvalds 553afeca340SKrishna Kumar static inline void tcp_mark_urg(struct tcp_sock *tp, int flags) 5541da177e4SLinus Torvalds { 55533f5f57eSIlpo Järvinen if (flags & MSG_OOB) 5561da177e4SLinus Torvalds tp->snd_up = tp->write_seq; 5571da177e4SLinus Torvalds } 5581da177e4SLinus Torvalds 5599e412ba7SIlpo Järvinen static inline void tcp_push(struct sock *sk, int flags, int mss_now, 5609e412ba7SIlpo Järvinen int nonagle) 5611da177e4SLinus Torvalds { 562afeca340SKrishna Kumar if (tcp_send_head(sk)) { 5639e412ba7SIlpo Järvinen struct tcp_sock *tp = tcp_sk(sk); 5649e412ba7SIlpo Järvinen 5651da177e4SLinus Torvalds if (!(flags & MSG_MORE) || forced_push(tp)) 566afeca340SKrishna Kumar tcp_mark_push(tp, tcp_write_queue_tail(sk)); 567afeca340SKrishna Kumar 568afeca340SKrishna Kumar tcp_mark_urg(tp, flags); 5699e412ba7SIlpo Järvinen __tcp_push_pending_frames(sk, mss_now, 5701da177e4SLinus Torvalds (flags & MSG_MORE) ? TCP_NAGLE_CORK : nonagle); 5711da177e4SLinus Torvalds } 5721da177e4SLinus Torvalds } 5731da177e4SLinus Torvalds 5746ff7751dSAdrian Bunk static int tcp_splice_data_recv(read_descriptor_t *rd_desc, struct sk_buff *skb, 5759c55e01cSJens Axboe unsigned int offset, size_t len) 5769c55e01cSJens Axboe { 5779c55e01cSJens Axboe struct tcp_splice_state *tss = rd_desc->arg.data; 57833966dd0SWilly Tarreau int ret; 5799c55e01cSJens Axboe 5809fa5fdf2SDimitris Michailidis ret = skb_splice_bits(skb, offset, tss->pipe, min(rd_desc->count, len), 5819fa5fdf2SDimitris Michailidis tss->flags); 58233966dd0SWilly Tarreau if (ret > 0) 58333966dd0SWilly Tarreau rd_desc->count -= ret; 58433966dd0SWilly Tarreau return ret; 5859c55e01cSJens Axboe } 5869c55e01cSJens Axboe 5879c55e01cSJens Axboe static int __tcp_splice_read(struct sock *sk, struct tcp_splice_state *tss) 5889c55e01cSJens Axboe { 5899c55e01cSJens Axboe /* Store TCP splice context information in read_descriptor_t. */ 5909c55e01cSJens Axboe read_descriptor_t rd_desc = { 5919c55e01cSJens Axboe .arg.data = tss, 59233966dd0SWilly Tarreau .count = tss->len, 5939c55e01cSJens Axboe }; 5949c55e01cSJens Axboe 5959c55e01cSJens Axboe return tcp_read_sock(sk, &rd_desc, tcp_splice_data_recv); 5969c55e01cSJens Axboe } 5979c55e01cSJens Axboe 5989c55e01cSJens Axboe /** 5999c55e01cSJens Axboe * tcp_splice_read - splice data from TCP socket to a pipe 6009c55e01cSJens Axboe * @sock: socket to splice from 6019c55e01cSJens Axboe * @ppos: position (not valid) 6029c55e01cSJens Axboe * @pipe: pipe to splice to 6039c55e01cSJens Axboe * @len: number of bytes to splice 6049c55e01cSJens Axboe * @flags: splice modifier flags 6059c55e01cSJens Axboe * 6069c55e01cSJens Axboe * Description: 6079c55e01cSJens Axboe * Will read pages from given socket and fill them into a pipe. 6089c55e01cSJens Axboe * 6099c55e01cSJens Axboe **/ 6109c55e01cSJens Axboe ssize_t tcp_splice_read(struct socket *sock, loff_t *ppos, 6119c55e01cSJens Axboe struct pipe_inode_info *pipe, size_t len, 6129c55e01cSJens Axboe unsigned int flags) 6139c55e01cSJens Axboe { 6149c55e01cSJens Axboe struct sock *sk = sock->sk; 6159c55e01cSJens Axboe struct tcp_splice_state tss = { 6169c55e01cSJens Axboe .pipe = pipe, 6179c55e01cSJens Axboe .len = len, 6189c55e01cSJens Axboe .flags = flags, 6199c55e01cSJens Axboe }; 6209c55e01cSJens Axboe long timeo; 6219c55e01cSJens Axboe ssize_t spliced; 6229c55e01cSJens Axboe int ret; 6239c55e01cSJens Axboe 6243a047bf8SChangli Gao sock_rps_record_flow(sk); 6259c55e01cSJens Axboe /* 6269c55e01cSJens Axboe * We can't seek on a socket input 6279c55e01cSJens Axboe */ 6289c55e01cSJens Axboe if (unlikely(*ppos)) 6299c55e01cSJens Axboe return -ESPIPE; 6309c55e01cSJens Axboe 6319c55e01cSJens Axboe ret = spliced = 0; 6329c55e01cSJens Axboe 6339c55e01cSJens Axboe lock_sock(sk); 6349c55e01cSJens Axboe 63542324c62SEric Dumazet timeo = sock_rcvtimeo(sk, sock->file->f_flags & O_NONBLOCK); 6369c55e01cSJens Axboe while (tss.len) { 6379c55e01cSJens Axboe ret = __tcp_splice_read(sk, &tss); 6389c55e01cSJens Axboe if (ret < 0) 6399c55e01cSJens Axboe break; 6409c55e01cSJens Axboe else if (!ret) { 6419c55e01cSJens Axboe if (spliced) 6429c55e01cSJens Axboe break; 6439c55e01cSJens Axboe if (sock_flag(sk, SOCK_DONE)) 6449c55e01cSJens Axboe break; 6459c55e01cSJens Axboe if (sk->sk_err) { 6469c55e01cSJens Axboe ret = sock_error(sk); 6479c55e01cSJens Axboe break; 6489c55e01cSJens Axboe } 6499c55e01cSJens Axboe if (sk->sk_shutdown & RCV_SHUTDOWN) 6509c55e01cSJens Axboe break; 6519c55e01cSJens Axboe if (sk->sk_state == TCP_CLOSE) { 6529c55e01cSJens Axboe /* 6539c55e01cSJens Axboe * This occurs when user tries to read 6549c55e01cSJens Axboe * from never connected socket. 6559c55e01cSJens Axboe */ 6569c55e01cSJens Axboe if (!sock_flag(sk, SOCK_DONE)) 6579c55e01cSJens Axboe ret = -ENOTCONN; 6589c55e01cSJens Axboe break; 6599c55e01cSJens Axboe } 6609c55e01cSJens Axboe if (!timeo) { 6619c55e01cSJens Axboe ret = -EAGAIN; 6629c55e01cSJens Axboe break; 6639c55e01cSJens Axboe } 6649c55e01cSJens Axboe sk_wait_data(sk, &timeo); 6659c55e01cSJens Axboe if (signal_pending(current)) { 6669c55e01cSJens Axboe ret = sock_intr_errno(timeo); 6679c55e01cSJens Axboe break; 6689c55e01cSJens Axboe } 6699c55e01cSJens Axboe continue; 6709c55e01cSJens Axboe } 6719c55e01cSJens Axboe tss.len -= ret; 6729c55e01cSJens Axboe spliced += ret; 6739c55e01cSJens Axboe 67433966dd0SWilly Tarreau if (!timeo) 67533966dd0SWilly Tarreau break; 6769c55e01cSJens Axboe release_sock(sk); 6779c55e01cSJens Axboe lock_sock(sk); 6789c55e01cSJens Axboe 6799c55e01cSJens Axboe if (sk->sk_err || sk->sk_state == TCP_CLOSE || 68033966dd0SWilly Tarreau (sk->sk_shutdown & RCV_SHUTDOWN) || 6819c55e01cSJens Axboe signal_pending(current)) 6829c55e01cSJens Axboe break; 6839c55e01cSJens Axboe } 6849c55e01cSJens Axboe 6859c55e01cSJens Axboe release_sock(sk); 6869c55e01cSJens Axboe 6879c55e01cSJens Axboe if (spliced) 6889c55e01cSJens Axboe return spliced; 6899c55e01cSJens Axboe 6909c55e01cSJens Axboe return ret; 6919c55e01cSJens Axboe } 6924bc2f18bSEric Dumazet EXPORT_SYMBOL(tcp_splice_read); 6939c55e01cSJens Axboe 694df97c708SPavel Emelyanov struct sk_buff *sk_stream_alloc_skb(struct sock *sk, int size, gfp_t gfp) 695f561d0f2SPavel Emelyanov { 696f561d0f2SPavel Emelyanov struct sk_buff *skb; 697f561d0f2SPavel Emelyanov 698f561d0f2SPavel Emelyanov /* The TCP header must be at least 32-bit aligned. */ 699f561d0f2SPavel Emelyanov size = ALIGN(size, 4); 700f561d0f2SPavel Emelyanov 701f561d0f2SPavel Emelyanov skb = alloc_skb_fclone(size + sk->sk_prot->max_header, gfp); 702f561d0f2SPavel Emelyanov if (skb) { 7033ab224beSHideo Aoki if (sk_wmem_schedule(sk, skb->truesize)) { 704f561d0f2SPavel Emelyanov /* 705f561d0f2SPavel Emelyanov * Make sure that we have exactly size bytes 706f561d0f2SPavel Emelyanov * available to the caller, no more, no less. 707f561d0f2SPavel Emelyanov */ 708f561d0f2SPavel Emelyanov skb_reserve(skb, skb_tailroom(skb) - size); 709f561d0f2SPavel Emelyanov return skb; 710f561d0f2SPavel Emelyanov } 711f561d0f2SPavel Emelyanov __kfree_skb(skb); 712f561d0f2SPavel Emelyanov } else { 7135c52ba17SPavel Emelyanov sk->sk_prot->enter_memory_pressure(sk); 714f561d0f2SPavel Emelyanov sk_stream_moderate_sndbuf(sk); 715f561d0f2SPavel Emelyanov } 716f561d0f2SPavel Emelyanov return NULL; 717f561d0f2SPavel Emelyanov } 718f561d0f2SPavel Emelyanov 7190c54b85fSIlpo Järvinen static unsigned int tcp_xmit_size_goal(struct sock *sk, u32 mss_now, 7200c54b85fSIlpo Järvinen int large_allowed) 7210c54b85fSIlpo Järvinen { 7220c54b85fSIlpo Järvinen struct tcp_sock *tp = tcp_sk(sk); 7232a3a041cSIlpo Järvinen u32 xmit_size_goal, old_size_goal; 7240c54b85fSIlpo Järvinen 7250c54b85fSIlpo Järvinen xmit_size_goal = mss_now; 7260c54b85fSIlpo Järvinen 7270c54b85fSIlpo Järvinen if (large_allowed && sk_can_gso(sk)) { 7280c54b85fSIlpo Järvinen xmit_size_goal = ((sk->sk_gso_max_size - 1) - 7290c54b85fSIlpo Järvinen inet_csk(sk)->icsk_af_ops->net_header_len - 7300c54b85fSIlpo Järvinen inet_csk(sk)->icsk_ext_hdr_len - 7310c54b85fSIlpo Järvinen tp->tcp_header_len); 7320c54b85fSIlpo Järvinen 7330c54b85fSIlpo Järvinen xmit_size_goal = tcp_bound_to_half_wnd(tp, xmit_size_goal); 7342a3a041cSIlpo Järvinen 7352a3a041cSIlpo Järvinen /* We try hard to avoid divides here */ 7362a3a041cSIlpo Järvinen old_size_goal = tp->xmit_size_goal_segs * mss_now; 7372a3a041cSIlpo Järvinen 7382a3a041cSIlpo Järvinen if (likely(old_size_goal <= xmit_size_goal && 7392a3a041cSIlpo Järvinen old_size_goal + mss_now > xmit_size_goal)) { 7402a3a041cSIlpo Järvinen xmit_size_goal = old_size_goal; 7412a3a041cSIlpo Järvinen } else { 7422a3a041cSIlpo Järvinen tp->xmit_size_goal_segs = xmit_size_goal / mss_now; 7432a3a041cSIlpo Järvinen xmit_size_goal = tp->xmit_size_goal_segs * mss_now; 7442a3a041cSIlpo Järvinen } 7450c54b85fSIlpo Järvinen } 7460c54b85fSIlpo Järvinen 747afece1c6SIlpo Järvinen return max(xmit_size_goal, mss_now); 7480c54b85fSIlpo Järvinen } 7490c54b85fSIlpo Järvinen 7500c54b85fSIlpo Järvinen static int tcp_send_mss(struct sock *sk, int *size_goal, int flags) 7510c54b85fSIlpo Järvinen { 7520c54b85fSIlpo Järvinen int mss_now; 7530c54b85fSIlpo Järvinen 7540c54b85fSIlpo Järvinen mss_now = tcp_current_mss(sk); 7550c54b85fSIlpo Järvinen *size_goal = tcp_xmit_size_goal(sk, mss_now, !(flags & MSG_OOB)); 7560c54b85fSIlpo Järvinen 7570c54b85fSIlpo Järvinen return mss_now; 7580c54b85fSIlpo Järvinen } 7590c54b85fSIlpo Järvinen 7601da177e4SLinus Torvalds static ssize_t do_tcp_sendpages(struct sock *sk, struct page **pages, int poffset, 7611da177e4SLinus Torvalds size_t psize, int flags) 7621da177e4SLinus Torvalds { 7631da177e4SLinus Torvalds struct tcp_sock *tp = tcp_sk(sk); 764c1b4a7e6SDavid S. Miller int mss_now, size_goal; 7651da177e4SLinus Torvalds int err; 7661da177e4SLinus Torvalds ssize_t copied; 7671da177e4SLinus Torvalds long timeo = sock_sndtimeo(sk, flags & MSG_DONTWAIT); 7681da177e4SLinus Torvalds 7691da177e4SLinus Torvalds /* Wait for a connection to finish. */ 7701da177e4SLinus Torvalds if ((1 << sk->sk_state) & ~(TCPF_ESTABLISHED | TCPF_CLOSE_WAIT)) 7711da177e4SLinus Torvalds if ((err = sk_stream_wait_connect(sk, &timeo)) != 0) 7721da177e4SLinus Torvalds goto out_err; 7731da177e4SLinus Torvalds 7741da177e4SLinus Torvalds clear_bit(SOCK_ASYNC_NOSPACE, &sk->sk_socket->flags); 7751da177e4SLinus Torvalds 7760c54b85fSIlpo Järvinen mss_now = tcp_send_mss(sk, &size_goal, flags); 7771da177e4SLinus Torvalds copied = 0; 7781da177e4SLinus Torvalds 7791da177e4SLinus Torvalds err = -EPIPE; 7801da177e4SLinus Torvalds if (sk->sk_err || (sk->sk_shutdown & SEND_SHUTDOWN)) 7810d6a775eSIlpo Järvinen goto out_err; 7821da177e4SLinus Torvalds 7831da177e4SLinus Torvalds while (psize > 0) { 784fe067e8aSDavid S. Miller struct sk_buff *skb = tcp_write_queue_tail(sk); 7851da177e4SLinus Torvalds struct page *page = pages[poffset / PAGE_SIZE]; 7861da177e4SLinus Torvalds int copy, i, can_coalesce; 7871da177e4SLinus Torvalds int offset = poffset % PAGE_SIZE; 7881da177e4SLinus Torvalds int size = min_t(size_t, psize, PAGE_SIZE - offset); 7891da177e4SLinus Torvalds 790fe067e8aSDavid S. Miller if (!tcp_send_head(sk) || (copy = size_goal - skb->len) <= 0) { 7911da177e4SLinus Torvalds new_segment: 7921da177e4SLinus Torvalds if (!sk_stream_memory_free(sk)) 7931da177e4SLinus Torvalds goto wait_for_sndbuf; 7941da177e4SLinus Torvalds 795df97c708SPavel Emelyanov skb = sk_stream_alloc_skb(sk, 0, sk->sk_allocation); 7961da177e4SLinus Torvalds if (!skb) 7971da177e4SLinus Torvalds goto wait_for_memory; 7981da177e4SLinus Torvalds 7999e412ba7SIlpo Järvinen skb_entail(sk, skb); 800c1b4a7e6SDavid S. Miller copy = size_goal; 8011da177e4SLinus Torvalds } 8021da177e4SLinus Torvalds 8031da177e4SLinus Torvalds if (copy > size) 8041da177e4SLinus Torvalds copy = size; 8051da177e4SLinus Torvalds 8061da177e4SLinus Torvalds i = skb_shinfo(skb)->nr_frags; 8071da177e4SLinus Torvalds can_coalesce = skb_can_coalesce(skb, i, page, offset); 8081da177e4SLinus Torvalds if (!can_coalesce && i >= MAX_SKB_FRAGS) { 8091da177e4SLinus Torvalds tcp_mark_push(tp, skb); 8101da177e4SLinus Torvalds goto new_segment; 8111da177e4SLinus Torvalds } 8123ab224beSHideo Aoki if (!sk_wmem_schedule(sk, copy)) 8131da177e4SLinus Torvalds goto wait_for_memory; 8141da177e4SLinus Torvalds 8151da177e4SLinus Torvalds if (can_coalesce) { 8161da177e4SLinus Torvalds skb_shinfo(skb)->frags[i - 1].size += copy; 8171da177e4SLinus Torvalds } else { 8181da177e4SLinus Torvalds get_page(page); 8191da177e4SLinus Torvalds skb_fill_page_desc(skb, i, page, offset, copy); 8201da177e4SLinus Torvalds } 8211da177e4SLinus Torvalds 8221da177e4SLinus Torvalds skb->len += copy; 8231da177e4SLinus Torvalds skb->data_len += copy; 8241da177e4SLinus Torvalds skb->truesize += copy; 8251da177e4SLinus Torvalds sk->sk_wmem_queued += copy; 8263ab224beSHideo Aoki sk_mem_charge(sk, copy); 82784fa7933SPatrick McHardy skb->ip_summed = CHECKSUM_PARTIAL; 8281da177e4SLinus Torvalds tp->write_seq += copy; 8291da177e4SLinus Torvalds TCP_SKB_CB(skb)->end_seq += copy; 8307967168cSHerbert Xu skb_shinfo(skb)->gso_segs = 0; 8311da177e4SLinus Torvalds 8321da177e4SLinus Torvalds if (!copied) 8334de075e0SEric Dumazet TCP_SKB_CB(skb)->tcp_flags &= ~TCPHDR_PSH; 8341da177e4SLinus Torvalds 8351da177e4SLinus Torvalds copied += copy; 8361da177e4SLinus Torvalds poffset += copy; 8371da177e4SLinus Torvalds if (!(psize -= copy)) 8381da177e4SLinus Torvalds goto out; 8391da177e4SLinus Torvalds 84069d15067SHerbert Xu if (skb->len < size_goal || (flags & MSG_OOB)) 8411da177e4SLinus Torvalds continue; 8421da177e4SLinus Torvalds 8431da177e4SLinus Torvalds if (forced_push(tp)) { 8441da177e4SLinus Torvalds tcp_mark_push(tp, skb); 8459e412ba7SIlpo Järvinen __tcp_push_pending_frames(sk, mss_now, TCP_NAGLE_PUSH); 846fe067e8aSDavid S. Miller } else if (skb == tcp_send_head(sk)) 8471da177e4SLinus Torvalds tcp_push_one(sk, mss_now); 8481da177e4SLinus Torvalds continue; 8491da177e4SLinus Torvalds 8501da177e4SLinus Torvalds wait_for_sndbuf: 8511da177e4SLinus Torvalds set_bit(SOCK_NOSPACE, &sk->sk_socket->flags); 8521da177e4SLinus Torvalds wait_for_memory: 8531da177e4SLinus Torvalds if (copied) 8549e412ba7SIlpo Järvinen tcp_push(sk, flags & ~MSG_MORE, mss_now, TCP_NAGLE_PUSH); 8551da177e4SLinus Torvalds 8561da177e4SLinus Torvalds if ((err = sk_stream_wait_memory(sk, &timeo)) != 0) 8571da177e4SLinus Torvalds goto do_error; 8581da177e4SLinus Torvalds 8590c54b85fSIlpo Järvinen mss_now = tcp_send_mss(sk, &size_goal, flags); 8601da177e4SLinus Torvalds } 8611da177e4SLinus Torvalds 8621da177e4SLinus Torvalds out: 8631da177e4SLinus Torvalds if (copied) 8649e412ba7SIlpo Järvinen tcp_push(sk, flags, mss_now, tp->nonagle); 8651da177e4SLinus Torvalds return copied; 8661da177e4SLinus Torvalds 8671da177e4SLinus Torvalds do_error: 8681da177e4SLinus Torvalds if (copied) 8691da177e4SLinus Torvalds goto out; 8701da177e4SLinus Torvalds out_err: 8711da177e4SLinus Torvalds return sk_stream_error(sk, flags, err); 8721da177e4SLinus Torvalds } 8731da177e4SLinus Torvalds 8747ba42910SChangli Gao int tcp_sendpage(struct sock *sk, struct page *page, int offset, 8751da177e4SLinus Torvalds size_t size, int flags) 8761da177e4SLinus Torvalds { 8771da177e4SLinus Torvalds ssize_t res; 8781da177e4SLinus Torvalds 8791da177e4SLinus Torvalds if (!(sk->sk_route_caps & NETIF_F_SG) || 8808648b305SHerbert Xu !(sk->sk_route_caps & NETIF_F_ALL_CSUM)) 8817ba42910SChangli Gao return sock_no_sendpage(sk->sk_socket, page, offset, size, 8827ba42910SChangli Gao flags); 8831da177e4SLinus Torvalds 8841da177e4SLinus Torvalds lock_sock(sk); 8851da177e4SLinus Torvalds res = do_tcp_sendpages(sk, &page, offset, size, flags); 8861da177e4SLinus Torvalds release_sock(sk); 8871da177e4SLinus Torvalds return res; 8881da177e4SLinus Torvalds } 8894bc2f18bSEric Dumazet EXPORT_SYMBOL(tcp_sendpage); 8901da177e4SLinus Torvalds 8911da177e4SLinus Torvalds #define TCP_PAGE(sk) (sk->sk_sndmsg_page) 8921da177e4SLinus Torvalds #define TCP_OFF(sk) (sk->sk_sndmsg_off) 8931da177e4SLinus Torvalds 894def87cf4SKrishna Kumar static inline int select_size(struct sock *sk, int sg) 8951da177e4SLinus Torvalds { 8969e412ba7SIlpo Järvinen struct tcp_sock *tp = tcp_sk(sk); 897c1b4a7e6SDavid S. Miller int tmp = tp->mss_cache; 8981da177e4SLinus Torvalds 899def87cf4SKrishna Kumar if (sg) { 900bcd76111SHerbert Xu if (sk_can_gso(sk)) 901c65f7f00SDavid S. Miller tmp = 0; 902b4e26f5eSDavid S. Miller else { 903b4e26f5eSDavid S. Miller int pgbreak = SKB_MAX_HEAD(MAX_TCP_HEADER); 904b4e26f5eSDavid S. Miller 905b4e26f5eSDavid S. Miller if (tmp >= pgbreak && 906b4e26f5eSDavid S. Miller tmp <= pgbreak + (MAX_SKB_FRAGS - 1) * PAGE_SIZE) 907b4e26f5eSDavid S. Miller tmp = pgbreak; 908b4e26f5eSDavid S. Miller } 909b4e26f5eSDavid S. Miller } 9101da177e4SLinus Torvalds 9111da177e4SLinus Torvalds return tmp; 9121da177e4SLinus Torvalds } 9131da177e4SLinus Torvalds 9147ba42910SChangli Gao int tcp_sendmsg(struct kiocb *iocb, struct sock *sk, struct msghdr *msg, 9151da177e4SLinus Torvalds size_t size) 9161da177e4SLinus Torvalds { 9171da177e4SLinus Torvalds struct iovec *iov; 9181da177e4SLinus Torvalds struct tcp_sock *tp = tcp_sk(sk); 9191da177e4SLinus Torvalds struct sk_buff *skb; 9201da177e4SLinus Torvalds int iovlen, flags; 921c1b4a7e6SDavid S. Miller int mss_now, size_goal; 922def87cf4SKrishna Kumar int sg, err, copied; 9231da177e4SLinus Torvalds long timeo; 9241da177e4SLinus Torvalds 9251da177e4SLinus Torvalds lock_sock(sk); 9261da177e4SLinus Torvalds 9271da177e4SLinus Torvalds flags = msg->msg_flags; 9281da177e4SLinus Torvalds timeo = sock_sndtimeo(sk, flags & MSG_DONTWAIT); 9291da177e4SLinus Torvalds 9301da177e4SLinus Torvalds /* Wait for a connection to finish. */ 9311da177e4SLinus Torvalds if ((1 << sk->sk_state) & ~(TCPF_ESTABLISHED | TCPF_CLOSE_WAIT)) 9321da177e4SLinus Torvalds if ((err = sk_stream_wait_connect(sk, &timeo)) != 0) 9331da177e4SLinus Torvalds goto out_err; 9341da177e4SLinus Torvalds 9351da177e4SLinus Torvalds /* This should be in poll */ 9361da177e4SLinus Torvalds clear_bit(SOCK_ASYNC_NOSPACE, &sk->sk_socket->flags); 9371da177e4SLinus Torvalds 9380c54b85fSIlpo Järvinen mss_now = tcp_send_mss(sk, &size_goal, flags); 9391da177e4SLinus Torvalds 9401da177e4SLinus Torvalds /* Ok commence sending. */ 9411da177e4SLinus Torvalds iovlen = msg->msg_iovlen; 9421da177e4SLinus Torvalds iov = msg->msg_iov; 9431da177e4SLinus Torvalds copied = 0; 9441da177e4SLinus Torvalds 9451da177e4SLinus Torvalds err = -EPIPE; 9461da177e4SLinus Torvalds if (sk->sk_err || (sk->sk_shutdown & SEND_SHUTDOWN)) 9470d6a775eSIlpo Järvinen goto out_err; 9481da177e4SLinus Torvalds 949def87cf4SKrishna Kumar sg = sk->sk_route_caps & NETIF_F_SG; 950def87cf4SKrishna Kumar 9511da177e4SLinus Torvalds while (--iovlen >= 0) { 95201db403cSDavid S. Miller size_t seglen = iov->iov_len; 9531da177e4SLinus Torvalds unsigned char __user *from = iov->iov_base; 9541da177e4SLinus Torvalds 9551da177e4SLinus Torvalds iov++; 9561da177e4SLinus Torvalds 9571da177e4SLinus Torvalds while (seglen > 0) { 9586828b92bSHerbert Xu int copy = 0; 9596828b92bSHerbert Xu int max = size_goal; 9601da177e4SLinus Torvalds 961fe067e8aSDavid S. Miller skb = tcp_write_queue_tail(sk); 9626828b92bSHerbert Xu if (tcp_send_head(sk)) { 9636828b92bSHerbert Xu if (skb->ip_summed == CHECKSUM_NONE) 9646828b92bSHerbert Xu max = mss_now; 9656828b92bSHerbert Xu copy = max - skb->len; 9666828b92bSHerbert Xu } 9671da177e4SLinus Torvalds 9686828b92bSHerbert Xu if (copy <= 0) { 9691da177e4SLinus Torvalds new_segment: 9701da177e4SLinus Torvalds /* Allocate new segment. If the interface is SG, 9711da177e4SLinus Torvalds * allocate skb fitting to single page. 9721da177e4SLinus Torvalds */ 9731da177e4SLinus Torvalds if (!sk_stream_memory_free(sk)) 9741da177e4SLinus Torvalds goto wait_for_sndbuf; 9751da177e4SLinus Torvalds 976def87cf4SKrishna Kumar skb = sk_stream_alloc_skb(sk, 977def87cf4SKrishna Kumar select_size(sk, sg), 978df97c708SPavel Emelyanov sk->sk_allocation); 9791da177e4SLinus Torvalds if (!skb) 9801da177e4SLinus Torvalds goto wait_for_memory; 9811da177e4SLinus Torvalds 9821da177e4SLinus Torvalds /* 9831da177e4SLinus Torvalds * Check whether we can use HW checksum. 9841da177e4SLinus Torvalds */ 9858648b305SHerbert Xu if (sk->sk_route_caps & NETIF_F_ALL_CSUM) 98684fa7933SPatrick McHardy skb->ip_summed = CHECKSUM_PARTIAL; 9871da177e4SLinus Torvalds 9889e412ba7SIlpo Järvinen skb_entail(sk, skb); 989c1b4a7e6SDavid S. Miller copy = size_goal; 9906828b92bSHerbert Xu max = size_goal; 9911da177e4SLinus Torvalds } 9921da177e4SLinus Torvalds 9931da177e4SLinus Torvalds /* Try to append data to the end of skb. */ 9941da177e4SLinus Torvalds if (copy > seglen) 9951da177e4SLinus Torvalds copy = seglen; 9961da177e4SLinus Torvalds 9971da177e4SLinus Torvalds /* Where to copy to? */ 9981da177e4SLinus Torvalds if (skb_tailroom(skb) > 0) { 9991da177e4SLinus Torvalds /* We have some space in skb head. Superb! */ 10001da177e4SLinus Torvalds if (copy > skb_tailroom(skb)) 10011da177e4SLinus Torvalds copy = skb_tailroom(skb); 1002c6e1a0d1STom Herbert err = skb_add_data_nocache(sk, skb, from, copy); 1003c6e1a0d1STom Herbert if (err) 10041da177e4SLinus Torvalds goto do_fault; 10051da177e4SLinus Torvalds } else { 10061da177e4SLinus Torvalds int merge = 0; 10071da177e4SLinus Torvalds int i = skb_shinfo(skb)->nr_frags; 10081da177e4SLinus Torvalds struct page *page = TCP_PAGE(sk); 10091da177e4SLinus Torvalds int off = TCP_OFF(sk); 10101da177e4SLinus Torvalds 10111da177e4SLinus Torvalds if (skb_can_coalesce(skb, i, page, off) && 10121da177e4SLinus Torvalds off != PAGE_SIZE) { 10131da177e4SLinus Torvalds /* We can extend the last page 10141da177e4SLinus Torvalds * fragment. */ 10151da177e4SLinus Torvalds merge = 1; 1016def87cf4SKrishna Kumar } else if (i == MAX_SKB_FRAGS || !sg) { 10171da177e4SLinus Torvalds /* Need to add new fragment and cannot 10181da177e4SLinus Torvalds * do this because interface is non-SG, 10191da177e4SLinus Torvalds * or because all the page slots are 10201da177e4SLinus Torvalds * busy. */ 10211da177e4SLinus Torvalds tcp_mark_push(tp, skb); 10221da177e4SLinus Torvalds goto new_segment; 10231da177e4SLinus Torvalds } else if (page) { 10241da177e4SLinus Torvalds if (off == PAGE_SIZE) { 10251da177e4SLinus Torvalds put_page(page); 10261da177e4SLinus Torvalds TCP_PAGE(sk) = page = NULL; 1027fb5f5e6eSHerbert Xu off = 0; 10281da177e4SLinus Torvalds } 1029ef015786SHerbert Xu } else 1030fb5f5e6eSHerbert Xu off = 0; 1031ef015786SHerbert Xu 1032ef015786SHerbert Xu if (copy > PAGE_SIZE - off) 1033ef015786SHerbert Xu copy = PAGE_SIZE - off; 1034ef015786SHerbert Xu 10353ab224beSHideo Aoki if (!sk_wmem_schedule(sk, copy)) 1036ef015786SHerbert Xu goto wait_for_memory; 10371da177e4SLinus Torvalds 10381da177e4SLinus Torvalds if (!page) { 10391da177e4SLinus Torvalds /* Allocate new cache page. */ 10401da177e4SLinus Torvalds if (!(page = sk_stream_alloc_page(sk))) 10411da177e4SLinus Torvalds goto wait_for_memory; 10421da177e4SLinus Torvalds } 10431da177e4SLinus Torvalds 10441da177e4SLinus Torvalds /* Time to copy data. We are close to 10451da177e4SLinus Torvalds * the end! */ 1046c6e1a0d1STom Herbert err = skb_copy_to_page_nocache(sk, from, skb, 1047c6e1a0d1STom Herbert page, off, copy); 10481da177e4SLinus Torvalds if (err) { 10491da177e4SLinus Torvalds /* If this page was new, give it to the 10501da177e4SLinus Torvalds * socket so it does not get leaked. 10511da177e4SLinus Torvalds */ 10521da177e4SLinus Torvalds if (!TCP_PAGE(sk)) { 10531da177e4SLinus Torvalds TCP_PAGE(sk) = page; 10541da177e4SLinus Torvalds TCP_OFF(sk) = 0; 10551da177e4SLinus Torvalds } 10561da177e4SLinus Torvalds goto do_error; 10571da177e4SLinus Torvalds } 10581da177e4SLinus Torvalds 10591da177e4SLinus Torvalds /* Update the skb. */ 10601da177e4SLinus Torvalds if (merge) { 10611da177e4SLinus Torvalds skb_shinfo(skb)->frags[i - 1].size += 10621da177e4SLinus Torvalds copy; 10631da177e4SLinus Torvalds } else { 10641da177e4SLinus Torvalds skb_fill_page_desc(skb, i, page, off, copy); 10651da177e4SLinus Torvalds if (TCP_PAGE(sk)) { 10661da177e4SLinus Torvalds get_page(page); 10671da177e4SLinus Torvalds } else if (off + copy < PAGE_SIZE) { 10681da177e4SLinus Torvalds get_page(page); 10691da177e4SLinus Torvalds TCP_PAGE(sk) = page; 10701da177e4SLinus Torvalds } 10711da177e4SLinus Torvalds } 10721da177e4SLinus Torvalds 10731da177e4SLinus Torvalds TCP_OFF(sk) = off + copy; 10741da177e4SLinus Torvalds } 10751da177e4SLinus Torvalds 10761da177e4SLinus Torvalds if (!copied) 10774de075e0SEric Dumazet TCP_SKB_CB(skb)->tcp_flags &= ~TCPHDR_PSH; 10781da177e4SLinus Torvalds 10791da177e4SLinus Torvalds tp->write_seq += copy; 10801da177e4SLinus Torvalds TCP_SKB_CB(skb)->end_seq += copy; 10817967168cSHerbert Xu skb_shinfo(skb)->gso_segs = 0; 10821da177e4SLinus Torvalds 10831da177e4SLinus Torvalds from += copy; 10841da177e4SLinus Torvalds copied += copy; 10851da177e4SLinus Torvalds if ((seglen -= copy) == 0 && iovlen == 0) 10861da177e4SLinus Torvalds goto out; 10871da177e4SLinus Torvalds 10886828b92bSHerbert Xu if (skb->len < max || (flags & MSG_OOB)) 10891da177e4SLinus Torvalds continue; 10901da177e4SLinus Torvalds 10911da177e4SLinus Torvalds if (forced_push(tp)) { 10921da177e4SLinus Torvalds tcp_mark_push(tp, skb); 10939e412ba7SIlpo Järvinen __tcp_push_pending_frames(sk, mss_now, TCP_NAGLE_PUSH); 1094fe067e8aSDavid S. Miller } else if (skb == tcp_send_head(sk)) 10951da177e4SLinus Torvalds tcp_push_one(sk, mss_now); 10961da177e4SLinus Torvalds continue; 10971da177e4SLinus Torvalds 10981da177e4SLinus Torvalds wait_for_sndbuf: 10991da177e4SLinus Torvalds set_bit(SOCK_NOSPACE, &sk->sk_socket->flags); 11001da177e4SLinus Torvalds wait_for_memory: 11011da177e4SLinus Torvalds if (copied) 11029e412ba7SIlpo Järvinen tcp_push(sk, flags & ~MSG_MORE, mss_now, TCP_NAGLE_PUSH); 11031da177e4SLinus Torvalds 11041da177e4SLinus Torvalds if ((err = sk_stream_wait_memory(sk, &timeo)) != 0) 11051da177e4SLinus Torvalds goto do_error; 11061da177e4SLinus Torvalds 11070c54b85fSIlpo Järvinen mss_now = tcp_send_mss(sk, &size_goal, flags); 11081da177e4SLinus Torvalds } 11091da177e4SLinus Torvalds } 11101da177e4SLinus Torvalds 11111da177e4SLinus Torvalds out: 11121da177e4SLinus Torvalds if (copied) 11139e412ba7SIlpo Järvinen tcp_push(sk, flags, mss_now, tp->nonagle); 11141da177e4SLinus Torvalds release_sock(sk); 11151da177e4SLinus Torvalds return copied; 11161da177e4SLinus Torvalds 11171da177e4SLinus Torvalds do_fault: 11181da177e4SLinus Torvalds if (!skb->len) { 1119fe067e8aSDavid S. Miller tcp_unlink_write_queue(skb, sk); 1120fe067e8aSDavid S. Miller /* It is the one place in all of TCP, except connection 1121fe067e8aSDavid S. Miller * reset, where we can be unlinking the send_head. 1122fe067e8aSDavid S. Miller */ 1123fe067e8aSDavid S. Miller tcp_check_send_head(sk, skb); 11243ab224beSHideo Aoki sk_wmem_free_skb(sk, skb); 11251da177e4SLinus Torvalds } 11261da177e4SLinus Torvalds 11271da177e4SLinus Torvalds do_error: 11281da177e4SLinus Torvalds if (copied) 11291da177e4SLinus Torvalds goto out; 11301da177e4SLinus Torvalds out_err: 11311da177e4SLinus Torvalds err = sk_stream_error(sk, flags, err); 11321da177e4SLinus Torvalds release_sock(sk); 11331da177e4SLinus Torvalds return err; 11341da177e4SLinus Torvalds } 11354bc2f18bSEric Dumazet EXPORT_SYMBOL(tcp_sendmsg); 11361da177e4SLinus Torvalds 11371da177e4SLinus Torvalds /* 11381da177e4SLinus Torvalds * Handle reading urgent data. BSD has very simple semantics for 11391da177e4SLinus Torvalds * this, no blocking and very strange errors 8) 11401da177e4SLinus Torvalds */ 11411da177e4SLinus Torvalds 1142377f0a08SRami Rosen static int tcp_recv_urg(struct sock *sk, struct msghdr *msg, int len, int flags) 11431da177e4SLinus Torvalds { 11441da177e4SLinus Torvalds struct tcp_sock *tp = tcp_sk(sk); 11451da177e4SLinus Torvalds 11461da177e4SLinus Torvalds /* No URG data to read. */ 11471da177e4SLinus Torvalds if (sock_flag(sk, SOCK_URGINLINE) || !tp->urg_data || 11481da177e4SLinus Torvalds tp->urg_data == TCP_URG_READ) 11491da177e4SLinus Torvalds return -EINVAL; /* Yes this is right ! */ 11501da177e4SLinus Torvalds 11511da177e4SLinus Torvalds if (sk->sk_state == TCP_CLOSE && !sock_flag(sk, SOCK_DONE)) 11521da177e4SLinus Torvalds return -ENOTCONN; 11531da177e4SLinus Torvalds 11541da177e4SLinus Torvalds if (tp->urg_data & TCP_URG_VALID) { 11551da177e4SLinus Torvalds int err = 0; 11561da177e4SLinus Torvalds char c = tp->urg_data; 11571da177e4SLinus Torvalds 11581da177e4SLinus Torvalds if (!(flags & MSG_PEEK)) 11591da177e4SLinus Torvalds tp->urg_data = TCP_URG_READ; 11601da177e4SLinus Torvalds 11611da177e4SLinus Torvalds /* Read urgent data. */ 11621da177e4SLinus Torvalds msg->msg_flags |= MSG_OOB; 11631da177e4SLinus Torvalds 11641da177e4SLinus Torvalds if (len > 0) { 11651da177e4SLinus Torvalds if (!(flags & MSG_TRUNC)) 11661da177e4SLinus Torvalds err = memcpy_toiovec(msg->msg_iov, &c, 1); 11671da177e4SLinus Torvalds len = 1; 11681da177e4SLinus Torvalds } else 11691da177e4SLinus Torvalds msg->msg_flags |= MSG_TRUNC; 11701da177e4SLinus Torvalds 11711da177e4SLinus Torvalds return err ? -EFAULT : len; 11721da177e4SLinus Torvalds } 11731da177e4SLinus Torvalds 11741da177e4SLinus Torvalds if (sk->sk_state == TCP_CLOSE || (sk->sk_shutdown & RCV_SHUTDOWN)) 11751da177e4SLinus Torvalds return 0; 11761da177e4SLinus Torvalds 11771da177e4SLinus Torvalds /* Fixed the recv(..., MSG_OOB) behaviour. BSD docs and 11781da177e4SLinus Torvalds * the available implementations agree in this case: 11791da177e4SLinus Torvalds * this call should never block, independent of the 11801da177e4SLinus Torvalds * blocking state of the socket. 11811da177e4SLinus Torvalds * Mike <pall@rz.uni-karlsruhe.de> 11821da177e4SLinus Torvalds */ 11831da177e4SLinus Torvalds return -EAGAIN; 11841da177e4SLinus Torvalds } 11851da177e4SLinus Torvalds 11861da177e4SLinus Torvalds /* Clean up the receive buffer for full frames taken by the user, 11871da177e4SLinus Torvalds * then send an ACK if necessary. COPIED is the number of bytes 11881da177e4SLinus Torvalds * tcp_recvmsg has given to the user so far, it speeds up the 11891da177e4SLinus Torvalds * calculation of whether or not we must ACK for the sake of 11901da177e4SLinus Torvalds * a window update. 11911da177e4SLinus Torvalds */ 11920e4b4992SChris Leech void tcp_cleanup_rbuf(struct sock *sk, int copied) 11931da177e4SLinus Torvalds { 11941da177e4SLinus Torvalds struct tcp_sock *tp = tcp_sk(sk); 11951da177e4SLinus Torvalds int time_to_ack = 0; 11961da177e4SLinus Torvalds 11971da177e4SLinus Torvalds #if TCP_DEBUG 11981da177e4SLinus Torvalds struct sk_buff *skb = skb_peek(&sk->sk_receive_queue); 11991da177e4SLinus Torvalds 1200d792c100SIlpo Järvinen WARN(skb && !before(tp->copied_seq, TCP_SKB_CB(skb)->end_seq), 12012af6fd8bSJoe Perches "cleanup rbuf bug: copied %X seq %X rcvnxt %X\n", 1202d792c100SIlpo Järvinen tp->copied_seq, TCP_SKB_CB(skb)->end_seq, tp->rcv_nxt); 12031da177e4SLinus Torvalds #endif 12041da177e4SLinus Torvalds 1205463c84b9SArnaldo Carvalho de Melo if (inet_csk_ack_scheduled(sk)) { 1206463c84b9SArnaldo Carvalho de Melo const struct inet_connection_sock *icsk = inet_csk(sk); 12071da177e4SLinus Torvalds /* Delayed ACKs frequently hit locked sockets during bulk 12081da177e4SLinus Torvalds * receive. */ 1209463c84b9SArnaldo Carvalho de Melo if (icsk->icsk_ack.blocked || 12101da177e4SLinus Torvalds /* Once-per-two-segments ACK was not sent by tcp_input.c */ 1211463c84b9SArnaldo Carvalho de Melo tp->rcv_nxt - tp->rcv_wup > icsk->icsk_ack.rcv_mss || 12121da177e4SLinus Torvalds /* 12131da177e4SLinus Torvalds * If this read emptied read buffer, we send ACK, if 12141da177e4SLinus Torvalds * connection is not bidirectional, user drained 12151da177e4SLinus Torvalds * receive buffer and there was a small segment 12161da177e4SLinus Torvalds * in queue. 12171da177e4SLinus Torvalds */ 12181ef9696cSAlexey Kuznetsov (copied > 0 && 12191ef9696cSAlexey Kuznetsov ((icsk->icsk_ack.pending & ICSK_ACK_PUSHED2) || 12201ef9696cSAlexey Kuznetsov ((icsk->icsk_ack.pending & ICSK_ACK_PUSHED) && 12211ef9696cSAlexey Kuznetsov !icsk->icsk_ack.pingpong)) && 12221ef9696cSAlexey Kuznetsov !atomic_read(&sk->sk_rmem_alloc))) 12231da177e4SLinus Torvalds time_to_ack = 1; 12241da177e4SLinus Torvalds } 12251da177e4SLinus Torvalds 12261da177e4SLinus Torvalds /* We send an ACK if we can now advertise a non-zero window 12271da177e4SLinus Torvalds * which has been raised "significantly". 12281da177e4SLinus Torvalds * 12291da177e4SLinus Torvalds * Even if window raised up to infinity, do not send window open ACK 12301da177e4SLinus Torvalds * in states, where we will not receive more. It is useless. 12311da177e4SLinus Torvalds */ 12321da177e4SLinus Torvalds if (copied > 0 && !time_to_ack && !(sk->sk_shutdown & RCV_SHUTDOWN)) { 12331da177e4SLinus Torvalds __u32 rcv_window_now = tcp_receive_window(tp); 12341da177e4SLinus Torvalds 12351da177e4SLinus Torvalds /* Optimize, __tcp_select_window() is not cheap. */ 12361da177e4SLinus Torvalds if (2*rcv_window_now <= tp->window_clamp) { 12371da177e4SLinus Torvalds __u32 new_window = __tcp_select_window(sk); 12381da177e4SLinus Torvalds 12391da177e4SLinus Torvalds /* Send ACK now, if this read freed lots of space 12401da177e4SLinus Torvalds * in our buffer. Certainly, new_window is new window. 12411da177e4SLinus Torvalds * We can advertise it now, if it is not less than current one. 12421da177e4SLinus Torvalds * "Lots" means "at least twice" here. 12431da177e4SLinus Torvalds */ 12441da177e4SLinus Torvalds if (new_window && new_window >= 2 * rcv_window_now) 12451da177e4SLinus Torvalds time_to_ack = 1; 12461da177e4SLinus Torvalds } 12471da177e4SLinus Torvalds } 12481da177e4SLinus Torvalds if (time_to_ack) 12491da177e4SLinus Torvalds tcp_send_ack(sk); 12501da177e4SLinus Torvalds } 12511da177e4SLinus Torvalds 12521da177e4SLinus Torvalds static void tcp_prequeue_process(struct sock *sk) 12531da177e4SLinus Torvalds { 12541da177e4SLinus Torvalds struct sk_buff *skb; 12551da177e4SLinus Torvalds struct tcp_sock *tp = tcp_sk(sk); 12561da177e4SLinus Torvalds 12576f67c817SPavel Emelyanov NET_INC_STATS_USER(sock_net(sk), LINUX_MIB_TCPPREQUEUED); 12581da177e4SLinus Torvalds 12591da177e4SLinus Torvalds /* RX process wants to run with disabled BHs, though it is not 12601da177e4SLinus Torvalds * necessary */ 12611da177e4SLinus Torvalds local_bh_disable(); 12621da177e4SLinus Torvalds while ((skb = __skb_dequeue(&tp->ucopy.prequeue)) != NULL) 1263c57943a1SPeter Zijlstra sk_backlog_rcv(sk, skb); 12641da177e4SLinus Torvalds local_bh_enable(); 12651da177e4SLinus Torvalds 12661da177e4SLinus Torvalds /* Clear memory counter. */ 12671da177e4SLinus Torvalds tp->ucopy.memory = 0; 12681da177e4SLinus Torvalds } 12691da177e4SLinus Torvalds 127073852e81SSteven J. Magnani #ifdef CONFIG_NET_DMA 127173852e81SSteven J. Magnani static void tcp_service_net_dma(struct sock *sk, bool wait) 127273852e81SSteven J. Magnani { 127373852e81SSteven J. Magnani dma_cookie_t done, used; 127473852e81SSteven J. Magnani dma_cookie_t last_issued; 127573852e81SSteven J. Magnani struct tcp_sock *tp = tcp_sk(sk); 127673852e81SSteven J. Magnani 127773852e81SSteven J. Magnani if (!tp->ucopy.dma_chan) 127873852e81SSteven J. Magnani return; 127973852e81SSteven J. Magnani 128073852e81SSteven J. Magnani last_issued = tp->ucopy.dma_cookie; 128173852e81SSteven J. Magnani dma_async_memcpy_issue_pending(tp->ucopy.dma_chan); 128273852e81SSteven J. Magnani 128373852e81SSteven J. Magnani do { 128473852e81SSteven J. Magnani if (dma_async_memcpy_complete(tp->ucopy.dma_chan, 128573852e81SSteven J. Magnani last_issued, &done, 128673852e81SSteven J. Magnani &used) == DMA_SUCCESS) { 128773852e81SSteven J. Magnani /* Safe to free early-copied skbs now */ 128873852e81SSteven J. Magnani __skb_queue_purge(&sk->sk_async_wait_queue); 128973852e81SSteven J. Magnani break; 129073852e81SSteven J. Magnani } else { 129173852e81SSteven J. Magnani struct sk_buff *skb; 129273852e81SSteven J. Magnani while ((skb = skb_peek(&sk->sk_async_wait_queue)) && 129373852e81SSteven J. Magnani (dma_async_is_complete(skb->dma_cookie, done, 129473852e81SSteven J. Magnani used) == DMA_SUCCESS)) { 129573852e81SSteven J. Magnani __skb_dequeue(&sk->sk_async_wait_queue); 129673852e81SSteven J. Magnani kfree_skb(skb); 129773852e81SSteven J. Magnani } 129873852e81SSteven J. Magnani } 129973852e81SSteven J. Magnani } while (wait); 130073852e81SSteven J. Magnani } 130173852e81SSteven J. Magnani #endif 130273852e81SSteven J. Magnani 13031da177e4SLinus Torvalds static inline struct sk_buff *tcp_recv_skb(struct sock *sk, u32 seq, u32 *off) 13041da177e4SLinus Torvalds { 13051da177e4SLinus Torvalds struct sk_buff *skb; 13061da177e4SLinus Torvalds u32 offset; 13071da177e4SLinus Torvalds 13081da177e4SLinus Torvalds skb_queue_walk(&sk->sk_receive_queue, skb) { 13091da177e4SLinus Torvalds offset = seq - TCP_SKB_CB(skb)->seq; 1310aa8223c7SArnaldo Carvalho de Melo if (tcp_hdr(skb)->syn) 13111da177e4SLinus Torvalds offset--; 1312aa8223c7SArnaldo Carvalho de Melo if (offset < skb->len || tcp_hdr(skb)->fin) { 13131da177e4SLinus Torvalds *off = offset; 13141da177e4SLinus Torvalds return skb; 13151da177e4SLinus Torvalds } 13161da177e4SLinus Torvalds } 13171da177e4SLinus Torvalds return NULL; 13181da177e4SLinus Torvalds } 13191da177e4SLinus Torvalds 13201da177e4SLinus Torvalds /* 13211da177e4SLinus Torvalds * This routine provides an alternative to tcp_recvmsg() for routines 13221da177e4SLinus Torvalds * that would like to handle copying from skbuffs directly in 'sendfile' 13231da177e4SLinus Torvalds * fashion. 13241da177e4SLinus Torvalds * Note: 13251da177e4SLinus Torvalds * - It is assumed that the socket was locked by the caller. 13261da177e4SLinus Torvalds * - The routine does not block. 13271da177e4SLinus Torvalds * - At present, there is no support for reading OOB data 13281da177e4SLinus Torvalds * or for 'peeking' the socket using this routine 13291da177e4SLinus Torvalds * (although both would be easy to implement). 13301da177e4SLinus Torvalds */ 13311da177e4SLinus Torvalds int tcp_read_sock(struct sock *sk, read_descriptor_t *desc, 13321da177e4SLinus Torvalds sk_read_actor_t recv_actor) 13331da177e4SLinus Torvalds { 13341da177e4SLinus Torvalds struct sk_buff *skb; 13351da177e4SLinus Torvalds struct tcp_sock *tp = tcp_sk(sk); 13361da177e4SLinus Torvalds u32 seq = tp->copied_seq; 13371da177e4SLinus Torvalds u32 offset; 13381da177e4SLinus Torvalds int copied = 0; 13391da177e4SLinus Torvalds 13401da177e4SLinus Torvalds if (sk->sk_state == TCP_LISTEN) 13411da177e4SLinus Torvalds return -ENOTCONN; 13421da177e4SLinus Torvalds while ((skb = tcp_recv_skb(sk, seq, &offset)) != NULL) { 13431da177e4SLinus Torvalds if (offset < skb->len) { 1344374e7b59SOctavian Purdila int used; 1345374e7b59SOctavian Purdila size_t len; 13461da177e4SLinus Torvalds 13471da177e4SLinus Torvalds len = skb->len - offset; 13481da177e4SLinus Torvalds /* Stop reading if we hit a patch of urgent data */ 13491da177e4SLinus Torvalds if (tp->urg_data) { 13501da177e4SLinus Torvalds u32 urg_offset = tp->urg_seq - seq; 13511da177e4SLinus Torvalds if (urg_offset < len) 13521da177e4SLinus Torvalds len = urg_offset; 13531da177e4SLinus Torvalds if (!len) 13541da177e4SLinus Torvalds break; 13551da177e4SLinus Torvalds } 13561da177e4SLinus Torvalds used = recv_actor(desc, skb, offset, len); 1357ddb61a57SJens Axboe if (used < 0) { 1358ddb61a57SJens Axboe if (!copied) 1359ddb61a57SJens Axboe copied = used; 1360ddb61a57SJens Axboe break; 1361ddb61a57SJens Axboe } else if (used <= len) { 13621da177e4SLinus Torvalds seq += used; 13631da177e4SLinus Torvalds copied += used; 13641da177e4SLinus Torvalds offset += used; 13651da177e4SLinus Torvalds } 1366293ad604SOctavian Purdila /* 1367293ad604SOctavian Purdila * If recv_actor drops the lock (e.g. TCP splice 1368293ad604SOctavian Purdila * receive) the skb pointer might be invalid when 1369293ad604SOctavian Purdila * getting here: tcp_collapse might have deleted it 1370293ad604SOctavian Purdila * while aggregating skbs from the socket queue. 1371293ad604SOctavian Purdila */ 1372293ad604SOctavian Purdila skb = tcp_recv_skb(sk, seq-1, &offset); 1373293ad604SOctavian Purdila if (!skb || (offset+1 != skb->len)) 13741da177e4SLinus Torvalds break; 13751da177e4SLinus Torvalds } 1376aa8223c7SArnaldo Carvalho de Melo if (tcp_hdr(skb)->fin) { 1377624d1164SChris Leech sk_eat_skb(sk, skb, 0); 13781da177e4SLinus Torvalds ++seq; 13791da177e4SLinus Torvalds break; 13801da177e4SLinus Torvalds } 1381624d1164SChris Leech sk_eat_skb(sk, skb, 0); 13821da177e4SLinus Torvalds if (!desc->count) 13831da177e4SLinus Torvalds break; 1384baff42abSSteven J. Magnani tp->copied_seq = seq; 13851da177e4SLinus Torvalds } 13861da177e4SLinus Torvalds tp->copied_seq = seq; 13871da177e4SLinus Torvalds 13881da177e4SLinus Torvalds tcp_rcv_space_adjust(sk); 13891da177e4SLinus Torvalds 13901da177e4SLinus Torvalds /* Clean up data we have read: This will do ACK frames. */ 1391ddb61a57SJens Axboe if (copied > 0) 13920e4b4992SChris Leech tcp_cleanup_rbuf(sk, copied); 13931da177e4SLinus Torvalds return copied; 13941da177e4SLinus Torvalds } 13954bc2f18bSEric Dumazet EXPORT_SYMBOL(tcp_read_sock); 13961da177e4SLinus Torvalds 13971da177e4SLinus Torvalds /* 13981da177e4SLinus Torvalds * This routine copies from a sock struct into the user buffer. 13991da177e4SLinus Torvalds * 14001da177e4SLinus Torvalds * Technical note: in 2.3 we work on _locked_ socket, so that 14011da177e4SLinus Torvalds * tricks with *seq access order and skb->users are not required. 14021da177e4SLinus Torvalds * Probably, code can be easily improved even more. 14031da177e4SLinus Torvalds */ 14041da177e4SLinus Torvalds 14051da177e4SLinus Torvalds int tcp_recvmsg(struct kiocb *iocb, struct sock *sk, struct msghdr *msg, 14061da177e4SLinus Torvalds size_t len, int nonblock, int flags, int *addr_len) 14071da177e4SLinus Torvalds { 14081da177e4SLinus Torvalds struct tcp_sock *tp = tcp_sk(sk); 14091da177e4SLinus Torvalds int copied = 0; 14101da177e4SLinus Torvalds u32 peek_seq; 14111da177e4SLinus Torvalds u32 *seq; 14121da177e4SLinus Torvalds unsigned long used; 14131da177e4SLinus Torvalds int err; 14141da177e4SLinus Torvalds int target; /* Read at least this many bytes */ 14151da177e4SLinus Torvalds long timeo; 14161da177e4SLinus Torvalds struct task_struct *user_recv = NULL; 14171a2449a8SChris Leech int copied_early = 0; 14182b1244a4SChris Leech struct sk_buff *skb; 141977527313SIlpo Järvinen u32 urg_hole = 0; 14201da177e4SLinus Torvalds 14211da177e4SLinus Torvalds lock_sock(sk); 14221da177e4SLinus Torvalds 14231da177e4SLinus Torvalds err = -ENOTCONN; 14241da177e4SLinus Torvalds if (sk->sk_state == TCP_LISTEN) 14251da177e4SLinus Torvalds goto out; 14261da177e4SLinus Torvalds 14271da177e4SLinus Torvalds timeo = sock_rcvtimeo(sk, nonblock); 14281da177e4SLinus Torvalds 14291da177e4SLinus Torvalds /* Urgent data needs to be handled specially. */ 14301da177e4SLinus Torvalds if (flags & MSG_OOB) 14311da177e4SLinus Torvalds goto recv_urg; 14321da177e4SLinus Torvalds 14331da177e4SLinus Torvalds seq = &tp->copied_seq; 14341da177e4SLinus Torvalds if (flags & MSG_PEEK) { 14351da177e4SLinus Torvalds peek_seq = tp->copied_seq; 14361da177e4SLinus Torvalds seq = &peek_seq; 14371da177e4SLinus Torvalds } 14381da177e4SLinus Torvalds 14391da177e4SLinus Torvalds target = sock_rcvlowat(sk, flags & MSG_WAITALL, len); 14401da177e4SLinus Torvalds 14411a2449a8SChris Leech #ifdef CONFIG_NET_DMA 14421a2449a8SChris Leech tp->ucopy.dma_chan = NULL; 14431a2449a8SChris Leech preempt_disable(); 14442b1244a4SChris Leech skb = skb_peek_tail(&sk->sk_receive_queue); 1445e00c5d8bSAndrew Morton { 1446e00c5d8bSAndrew Morton int available = 0; 1447e00c5d8bSAndrew Morton 14482b1244a4SChris Leech if (skb) 14492b1244a4SChris Leech available = TCP_SKB_CB(skb)->seq + skb->len - (*seq); 14502b1244a4SChris Leech if ((available < target) && 14512b1244a4SChris Leech (len > sysctl_tcp_dma_copybreak) && !(flags & MSG_PEEK) && 1452e00c5d8bSAndrew Morton !sysctl_tcp_low_latency && 1453f67b4599SDan Williams dma_find_channel(DMA_MEMCPY)) { 14541a2449a8SChris Leech preempt_enable_no_resched(); 1455e00c5d8bSAndrew Morton tp->ucopy.pinned_list = 1456e00c5d8bSAndrew Morton dma_pin_iovec_pages(msg->msg_iov, len); 1457e00c5d8bSAndrew Morton } else { 14581a2449a8SChris Leech preempt_enable_no_resched(); 1459e00c5d8bSAndrew Morton } 1460e00c5d8bSAndrew Morton } 14611a2449a8SChris Leech #endif 14621a2449a8SChris Leech 14631da177e4SLinus Torvalds do { 14641da177e4SLinus Torvalds u32 offset; 14651da177e4SLinus Torvalds 14661da177e4SLinus Torvalds /* Are we at urgent data? Stop if we have read anything or have SIGURG pending. */ 14671da177e4SLinus Torvalds if (tp->urg_data && tp->urg_seq == *seq) { 14681da177e4SLinus Torvalds if (copied) 14691da177e4SLinus Torvalds break; 14701da177e4SLinus Torvalds if (signal_pending(current)) { 14711da177e4SLinus Torvalds copied = timeo ? sock_intr_errno(timeo) : -EAGAIN; 14721da177e4SLinus Torvalds break; 14731da177e4SLinus Torvalds } 14741da177e4SLinus Torvalds } 14751da177e4SLinus Torvalds 14761da177e4SLinus Torvalds /* Next get a buffer. */ 14771da177e4SLinus Torvalds 147891521944SDavid S. Miller skb_queue_walk(&sk->sk_receive_queue, skb) { 14791da177e4SLinus Torvalds /* Now that we have two receive queues this 14801da177e4SLinus Torvalds * shouldn't happen. 14811da177e4SLinus Torvalds */ 1482d792c100SIlpo Järvinen if (WARN(before(*seq, TCP_SKB_CB(skb)->seq), 14832af6fd8bSJoe Perches "recvmsg bug: copied %X seq %X rcvnxt %X fl %X\n", 14842af6fd8bSJoe Perches *seq, TCP_SKB_CB(skb)->seq, tp->rcv_nxt, 1485d792c100SIlpo Järvinen flags)) 14861da177e4SLinus Torvalds break; 1487d792c100SIlpo Järvinen 14881da177e4SLinus Torvalds offset = *seq - TCP_SKB_CB(skb)->seq; 1489aa8223c7SArnaldo Carvalho de Melo if (tcp_hdr(skb)->syn) 14901da177e4SLinus Torvalds offset--; 14911da177e4SLinus Torvalds if (offset < skb->len) 14921da177e4SLinus Torvalds goto found_ok_skb; 1493aa8223c7SArnaldo Carvalho de Melo if (tcp_hdr(skb)->fin) 14941da177e4SLinus Torvalds goto found_fin_ok; 14952af6fd8bSJoe Perches WARN(!(flags & MSG_PEEK), 14962af6fd8bSJoe Perches "recvmsg bug 2: copied %X seq %X rcvnxt %X fl %X\n", 14972af6fd8bSJoe Perches *seq, TCP_SKB_CB(skb)->seq, tp->rcv_nxt, flags); 149891521944SDavid S. Miller } 14991da177e4SLinus Torvalds 15001da177e4SLinus Torvalds /* Well, if we have backlog, try to process it now yet. */ 15011da177e4SLinus Torvalds 15021da177e4SLinus Torvalds if (copied >= target && !sk->sk_backlog.tail) 15031da177e4SLinus Torvalds break; 15041da177e4SLinus Torvalds 15051da177e4SLinus Torvalds if (copied) { 15061da177e4SLinus Torvalds if (sk->sk_err || 15071da177e4SLinus Torvalds sk->sk_state == TCP_CLOSE || 15081da177e4SLinus Torvalds (sk->sk_shutdown & RCV_SHUTDOWN) || 15091da177e4SLinus Torvalds !timeo || 1510518a09efSDavid S. Miller signal_pending(current)) 15111da177e4SLinus Torvalds break; 15121da177e4SLinus Torvalds } else { 15131da177e4SLinus Torvalds if (sock_flag(sk, SOCK_DONE)) 15141da177e4SLinus Torvalds break; 15151da177e4SLinus Torvalds 15161da177e4SLinus Torvalds if (sk->sk_err) { 15171da177e4SLinus Torvalds copied = sock_error(sk); 15181da177e4SLinus Torvalds break; 15191da177e4SLinus Torvalds } 15201da177e4SLinus Torvalds 15211da177e4SLinus Torvalds if (sk->sk_shutdown & RCV_SHUTDOWN) 15221da177e4SLinus Torvalds break; 15231da177e4SLinus Torvalds 15241da177e4SLinus Torvalds if (sk->sk_state == TCP_CLOSE) { 15251da177e4SLinus Torvalds if (!sock_flag(sk, SOCK_DONE)) { 15261da177e4SLinus Torvalds /* This occurs when user tries to read 15271da177e4SLinus Torvalds * from never connected socket. 15281da177e4SLinus Torvalds */ 15291da177e4SLinus Torvalds copied = -ENOTCONN; 15301da177e4SLinus Torvalds break; 15311da177e4SLinus Torvalds } 15321da177e4SLinus Torvalds break; 15331da177e4SLinus Torvalds } 15341da177e4SLinus Torvalds 15351da177e4SLinus Torvalds if (!timeo) { 15361da177e4SLinus Torvalds copied = -EAGAIN; 15371da177e4SLinus Torvalds break; 15381da177e4SLinus Torvalds } 15391da177e4SLinus Torvalds 15401da177e4SLinus Torvalds if (signal_pending(current)) { 15411da177e4SLinus Torvalds copied = sock_intr_errno(timeo); 15421da177e4SLinus Torvalds break; 15431da177e4SLinus Torvalds } 15441da177e4SLinus Torvalds } 15451da177e4SLinus Torvalds 15460e4b4992SChris Leech tcp_cleanup_rbuf(sk, copied); 15471da177e4SLinus Torvalds 15487df55125SDavid S. Miller if (!sysctl_tcp_low_latency && tp->ucopy.task == user_recv) { 15491da177e4SLinus Torvalds /* Install new reader */ 15501da177e4SLinus Torvalds if (!user_recv && !(flags & (MSG_TRUNC | MSG_PEEK))) { 15511da177e4SLinus Torvalds user_recv = current; 15521da177e4SLinus Torvalds tp->ucopy.task = user_recv; 15531da177e4SLinus Torvalds tp->ucopy.iov = msg->msg_iov; 15541da177e4SLinus Torvalds } 15551da177e4SLinus Torvalds 15561da177e4SLinus Torvalds tp->ucopy.len = len; 15571da177e4SLinus Torvalds 1558547b792cSIlpo Järvinen WARN_ON(tp->copied_seq != tp->rcv_nxt && 1559547b792cSIlpo Järvinen !(flags & (MSG_PEEK | MSG_TRUNC))); 15601da177e4SLinus Torvalds 15611da177e4SLinus Torvalds /* Ugly... If prequeue is not empty, we have to 15621da177e4SLinus Torvalds * process it before releasing socket, otherwise 15631da177e4SLinus Torvalds * order will be broken at second iteration. 15641da177e4SLinus Torvalds * More elegant solution is required!!! 15651da177e4SLinus Torvalds * 15661da177e4SLinus Torvalds * Look: we have the following (pseudo)queues: 15671da177e4SLinus Torvalds * 15681da177e4SLinus Torvalds * 1. packets in flight 15691da177e4SLinus Torvalds * 2. backlog 15701da177e4SLinus Torvalds * 3. prequeue 15711da177e4SLinus Torvalds * 4. receive_queue 15721da177e4SLinus Torvalds * 15731da177e4SLinus Torvalds * Each queue can be processed only if the next ones 15741da177e4SLinus Torvalds * are empty. At this point we have empty receive_queue. 15751da177e4SLinus Torvalds * But prequeue _can_ be not empty after 2nd iteration, 15761da177e4SLinus Torvalds * when we jumped to start of loop because backlog 15771da177e4SLinus Torvalds * processing added something to receive_queue. 15781da177e4SLinus Torvalds * We cannot release_sock(), because backlog contains 15791da177e4SLinus Torvalds * packets arrived _after_ prequeued ones. 15801da177e4SLinus Torvalds * 15811da177e4SLinus Torvalds * Shortly, algorithm is clear --- to process all 15821da177e4SLinus Torvalds * the queues in order. We could make it more directly, 15831da177e4SLinus Torvalds * requeueing packets from backlog to prequeue, if 15841da177e4SLinus Torvalds * is not empty. It is more elegant, but eats cycles, 15851da177e4SLinus Torvalds * unfortunately. 15861da177e4SLinus Torvalds */ 1587b03efcfbSDavid S. Miller if (!skb_queue_empty(&tp->ucopy.prequeue)) 15881da177e4SLinus Torvalds goto do_prequeue; 15891da177e4SLinus Torvalds 15901da177e4SLinus Torvalds /* __ Set realtime policy in scheduler __ */ 15911da177e4SLinus Torvalds } 15921da177e4SLinus Torvalds 159373852e81SSteven J. Magnani #ifdef CONFIG_NET_DMA 159473852e81SSteven J. Magnani if (tp->ucopy.dma_chan) 159573852e81SSteven J. Magnani dma_async_memcpy_issue_pending(tp->ucopy.dma_chan); 159673852e81SSteven J. Magnani #endif 15971da177e4SLinus Torvalds if (copied >= target) { 15981da177e4SLinus Torvalds /* Do not sleep, just process backlog. */ 15991da177e4SLinus Torvalds release_sock(sk); 16001da177e4SLinus Torvalds lock_sock(sk); 16011da177e4SLinus Torvalds } else 16021da177e4SLinus Torvalds sk_wait_data(sk, &timeo); 16031da177e4SLinus Torvalds 16041a2449a8SChris Leech #ifdef CONFIG_NET_DMA 160573852e81SSteven J. Magnani tcp_service_net_dma(sk, false); /* Don't block */ 16061a2449a8SChris Leech tp->ucopy.wakeup = 0; 16071a2449a8SChris Leech #endif 16081a2449a8SChris Leech 16091da177e4SLinus Torvalds if (user_recv) { 16101da177e4SLinus Torvalds int chunk; 16111da177e4SLinus Torvalds 16121da177e4SLinus Torvalds /* __ Restore normal policy in scheduler __ */ 16131da177e4SLinus Torvalds 16141da177e4SLinus Torvalds if ((chunk = len - tp->ucopy.len) != 0) { 1615ed88098eSPavel Emelyanov NET_ADD_STATS_USER(sock_net(sk), LINUX_MIB_TCPDIRECTCOPYFROMBACKLOG, chunk); 16161da177e4SLinus Torvalds len -= chunk; 16171da177e4SLinus Torvalds copied += chunk; 16181da177e4SLinus Torvalds } 16191da177e4SLinus Torvalds 16201da177e4SLinus Torvalds if (tp->rcv_nxt == tp->copied_seq && 1621b03efcfbSDavid S. Miller !skb_queue_empty(&tp->ucopy.prequeue)) { 16221da177e4SLinus Torvalds do_prequeue: 16231da177e4SLinus Torvalds tcp_prequeue_process(sk); 16241da177e4SLinus Torvalds 16251da177e4SLinus Torvalds if ((chunk = len - tp->ucopy.len) != 0) { 1626ed88098eSPavel Emelyanov NET_ADD_STATS_USER(sock_net(sk), LINUX_MIB_TCPDIRECTCOPYFROMPREQUEUE, chunk); 16271da177e4SLinus Torvalds len -= chunk; 16281da177e4SLinus Torvalds copied += chunk; 16291da177e4SLinus Torvalds } 16301da177e4SLinus Torvalds } 16311da177e4SLinus Torvalds } 163277527313SIlpo Järvinen if ((flags & MSG_PEEK) && 163377527313SIlpo Järvinen (peek_seq - copied - urg_hole != tp->copied_seq)) { 16341da177e4SLinus Torvalds if (net_ratelimit()) 16351da177e4SLinus Torvalds printk(KERN_DEBUG "TCP(%s:%d): Application bug, race in MSG_PEEK.\n", 1636ba25f9dcSPavel Emelyanov current->comm, task_pid_nr(current)); 16371da177e4SLinus Torvalds peek_seq = tp->copied_seq; 16381da177e4SLinus Torvalds } 16391da177e4SLinus Torvalds continue; 16401da177e4SLinus Torvalds 16411da177e4SLinus Torvalds found_ok_skb: 16421da177e4SLinus Torvalds /* Ok so how much can we use? */ 16431da177e4SLinus Torvalds used = skb->len - offset; 16441da177e4SLinus Torvalds if (len < used) 16451da177e4SLinus Torvalds used = len; 16461da177e4SLinus Torvalds 16471da177e4SLinus Torvalds /* Do we have urgent data here? */ 16481da177e4SLinus Torvalds if (tp->urg_data) { 16491da177e4SLinus Torvalds u32 urg_offset = tp->urg_seq - *seq; 16501da177e4SLinus Torvalds if (urg_offset < used) { 16511da177e4SLinus Torvalds if (!urg_offset) { 16521da177e4SLinus Torvalds if (!sock_flag(sk, SOCK_URGINLINE)) { 16531da177e4SLinus Torvalds ++*seq; 165477527313SIlpo Järvinen urg_hole++; 16551da177e4SLinus Torvalds offset++; 16561da177e4SLinus Torvalds used--; 16571da177e4SLinus Torvalds if (!used) 16581da177e4SLinus Torvalds goto skip_copy; 16591da177e4SLinus Torvalds } 16601da177e4SLinus Torvalds } else 16611da177e4SLinus Torvalds used = urg_offset; 16621da177e4SLinus Torvalds } 16631da177e4SLinus Torvalds } 16641da177e4SLinus Torvalds 16651da177e4SLinus Torvalds if (!(flags & MSG_TRUNC)) { 16661a2449a8SChris Leech #ifdef CONFIG_NET_DMA 16671a2449a8SChris Leech if (!tp->ucopy.dma_chan && tp->ucopy.pinned_list) 1668f67b4599SDan Williams tp->ucopy.dma_chan = dma_find_channel(DMA_MEMCPY); 16691a2449a8SChris Leech 16701a2449a8SChris Leech if (tp->ucopy.dma_chan) { 16711a2449a8SChris Leech tp->ucopy.dma_cookie = dma_skb_copy_datagram_iovec( 16721a2449a8SChris Leech tp->ucopy.dma_chan, skb, offset, 16731a2449a8SChris Leech msg->msg_iov, used, 16741a2449a8SChris Leech tp->ucopy.pinned_list); 16751a2449a8SChris Leech 16761a2449a8SChris Leech if (tp->ucopy.dma_cookie < 0) { 16771a2449a8SChris Leech 16781a2449a8SChris Leech printk(KERN_ALERT "dma_cookie < 0\n"); 16791a2449a8SChris Leech 16801a2449a8SChris Leech /* Exception. Bailout! */ 16811a2449a8SChris Leech if (!copied) 16821a2449a8SChris Leech copied = -EFAULT; 16831a2449a8SChris Leech break; 16841a2449a8SChris Leech } 168573852e81SSteven J. Magnani 168673852e81SSteven J. Magnani dma_async_memcpy_issue_pending(tp->ucopy.dma_chan); 168773852e81SSteven J. Magnani 16881a2449a8SChris Leech if ((offset + used) == skb->len) 16891a2449a8SChris Leech copied_early = 1; 16901a2449a8SChris Leech 16911a2449a8SChris Leech } else 16921a2449a8SChris Leech #endif 16931a2449a8SChris Leech { 16941da177e4SLinus Torvalds err = skb_copy_datagram_iovec(skb, offset, 16951da177e4SLinus Torvalds msg->msg_iov, used); 16961da177e4SLinus Torvalds if (err) { 16971da177e4SLinus Torvalds /* Exception. Bailout! */ 16981da177e4SLinus Torvalds if (!copied) 16991da177e4SLinus Torvalds copied = -EFAULT; 17001da177e4SLinus Torvalds break; 17011da177e4SLinus Torvalds } 17021da177e4SLinus Torvalds } 17031a2449a8SChris Leech } 17041da177e4SLinus Torvalds 17051da177e4SLinus Torvalds *seq += used; 17061da177e4SLinus Torvalds copied += used; 17071da177e4SLinus Torvalds len -= used; 17081da177e4SLinus Torvalds 17091da177e4SLinus Torvalds tcp_rcv_space_adjust(sk); 17101da177e4SLinus Torvalds 17111da177e4SLinus Torvalds skip_copy: 17121da177e4SLinus Torvalds if (tp->urg_data && after(tp->copied_seq, tp->urg_seq)) { 17131da177e4SLinus Torvalds tp->urg_data = 0; 17149e412ba7SIlpo Järvinen tcp_fast_path_check(sk); 17151da177e4SLinus Torvalds } 17161da177e4SLinus Torvalds if (used + offset < skb->len) 17171da177e4SLinus Torvalds continue; 17181da177e4SLinus Torvalds 1719aa8223c7SArnaldo Carvalho de Melo if (tcp_hdr(skb)->fin) 17201da177e4SLinus Torvalds goto found_fin_ok; 17211a2449a8SChris Leech if (!(flags & MSG_PEEK)) { 17221a2449a8SChris Leech sk_eat_skb(sk, skb, copied_early); 17231a2449a8SChris Leech copied_early = 0; 17241a2449a8SChris Leech } 17251da177e4SLinus Torvalds continue; 17261da177e4SLinus Torvalds 17271da177e4SLinus Torvalds found_fin_ok: 17281da177e4SLinus Torvalds /* Process the FIN. */ 17291da177e4SLinus Torvalds ++*seq; 17301a2449a8SChris Leech if (!(flags & MSG_PEEK)) { 17311a2449a8SChris Leech sk_eat_skb(sk, skb, copied_early); 17321a2449a8SChris Leech copied_early = 0; 17331a2449a8SChris Leech } 17341da177e4SLinus Torvalds break; 17351da177e4SLinus Torvalds } while (len > 0); 17361da177e4SLinus Torvalds 17371da177e4SLinus Torvalds if (user_recv) { 1738b03efcfbSDavid S. Miller if (!skb_queue_empty(&tp->ucopy.prequeue)) { 17391da177e4SLinus Torvalds int chunk; 17401da177e4SLinus Torvalds 17411da177e4SLinus Torvalds tp->ucopy.len = copied > 0 ? len : 0; 17421da177e4SLinus Torvalds 17431da177e4SLinus Torvalds tcp_prequeue_process(sk); 17441da177e4SLinus Torvalds 17451da177e4SLinus Torvalds if (copied > 0 && (chunk = len - tp->ucopy.len) != 0) { 1746ed88098eSPavel Emelyanov NET_ADD_STATS_USER(sock_net(sk), LINUX_MIB_TCPDIRECTCOPYFROMPREQUEUE, chunk); 17471da177e4SLinus Torvalds len -= chunk; 17481da177e4SLinus Torvalds copied += chunk; 17491da177e4SLinus Torvalds } 17501da177e4SLinus Torvalds } 17511da177e4SLinus Torvalds 17521da177e4SLinus Torvalds tp->ucopy.task = NULL; 17531da177e4SLinus Torvalds tp->ucopy.len = 0; 17541da177e4SLinus Torvalds } 17551da177e4SLinus Torvalds 17561a2449a8SChris Leech #ifdef CONFIG_NET_DMA 175773852e81SSteven J. Magnani tcp_service_net_dma(sk, true); /* Wait for queue to drain */ 17581a2449a8SChris Leech tp->ucopy.dma_chan = NULL; 175973852e81SSteven J. Magnani 17601a2449a8SChris Leech if (tp->ucopy.pinned_list) { 17611a2449a8SChris Leech dma_unpin_iovec_pages(tp->ucopy.pinned_list); 17621a2449a8SChris Leech tp->ucopy.pinned_list = NULL; 17631a2449a8SChris Leech } 17641a2449a8SChris Leech #endif 17651a2449a8SChris Leech 17661da177e4SLinus Torvalds /* According to UNIX98, msg_name/msg_namelen are ignored 17671da177e4SLinus Torvalds * on connected socket. I was just happy when found this 8) --ANK 17681da177e4SLinus Torvalds */ 17691da177e4SLinus Torvalds 17701da177e4SLinus Torvalds /* Clean up data we have read: This will do ACK frames. */ 17710e4b4992SChris Leech tcp_cleanup_rbuf(sk, copied); 17721da177e4SLinus Torvalds 17731da177e4SLinus Torvalds release_sock(sk); 17741da177e4SLinus Torvalds return copied; 17751da177e4SLinus Torvalds 17761da177e4SLinus Torvalds out: 17771da177e4SLinus Torvalds release_sock(sk); 17781da177e4SLinus Torvalds return err; 17791da177e4SLinus Torvalds 17801da177e4SLinus Torvalds recv_urg: 1781377f0a08SRami Rosen err = tcp_recv_urg(sk, msg, len, flags); 17821da177e4SLinus Torvalds goto out; 17831da177e4SLinus Torvalds } 17844bc2f18bSEric Dumazet EXPORT_SYMBOL(tcp_recvmsg); 17851da177e4SLinus Torvalds 1786490d5046SIlpo Järvinen void tcp_set_state(struct sock *sk, int state) 1787490d5046SIlpo Järvinen { 1788490d5046SIlpo Järvinen int oldstate = sk->sk_state; 1789490d5046SIlpo Järvinen 1790490d5046SIlpo Järvinen switch (state) { 1791490d5046SIlpo Järvinen case TCP_ESTABLISHED: 1792490d5046SIlpo Järvinen if (oldstate != TCP_ESTABLISHED) 179381cc8a75SPavel Emelyanov TCP_INC_STATS(sock_net(sk), TCP_MIB_CURRESTAB); 1794490d5046SIlpo Järvinen break; 1795490d5046SIlpo Järvinen 1796490d5046SIlpo Järvinen case TCP_CLOSE: 1797490d5046SIlpo Järvinen if (oldstate == TCP_CLOSE_WAIT || oldstate == TCP_ESTABLISHED) 179881cc8a75SPavel Emelyanov TCP_INC_STATS(sock_net(sk), TCP_MIB_ESTABRESETS); 1799490d5046SIlpo Järvinen 1800490d5046SIlpo Järvinen sk->sk_prot->unhash(sk); 1801490d5046SIlpo Järvinen if (inet_csk(sk)->icsk_bind_hash && 1802490d5046SIlpo Järvinen !(sk->sk_userlocks & SOCK_BINDPORT_LOCK)) 1803ab1e0a13SArnaldo Carvalho de Melo inet_put_port(sk); 1804490d5046SIlpo Järvinen /* fall through */ 1805490d5046SIlpo Järvinen default: 1806490d5046SIlpo Järvinen if (oldstate == TCP_ESTABLISHED) 180774688e48SPavel Emelyanov TCP_DEC_STATS(sock_net(sk), TCP_MIB_CURRESTAB); 1808490d5046SIlpo Järvinen } 1809490d5046SIlpo Järvinen 1810490d5046SIlpo Järvinen /* Change state AFTER socket is unhashed to avoid closed 1811490d5046SIlpo Järvinen * socket sitting in hash tables. 1812490d5046SIlpo Järvinen */ 1813490d5046SIlpo Järvinen sk->sk_state = state; 1814490d5046SIlpo Järvinen 1815490d5046SIlpo Järvinen #ifdef STATE_TRACE 1816490d5046SIlpo Järvinen SOCK_DEBUG(sk, "TCP sk=%p, State %s -> %s\n", sk, statename[oldstate], statename[state]); 1817490d5046SIlpo Järvinen #endif 1818490d5046SIlpo Järvinen } 1819490d5046SIlpo Järvinen EXPORT_SYMBOL_GPL(tcp_set_state); 1820490d5046SIlpo Järvinen 18211da177e4SLinus Torvalds /* 18221da177e4SLinus Torvalds * State processing on a close. This implements the state shift for 18231da177e4SLinus Torvalds * sending our FIN frame. Note that we only send a FIN for some 18241da177e4SLinus Torvalds * states. A shutdown() may have already sent the FIN, or we may be 18251da177e4SLinus Torvalds * closed. 18261da177e4SLinus Torvalds */ 18271da177e4SLinus Torvalds 18289b5b5cffSArjan van de Ven static const unsigned char new_state[16] = { 18291da177e4SLinus Torvalds /* current state: new state: action: */ 18301da177e4SLinus Torvalds /* (Invalid) */ TCP_CLOSE, 18311da177e4SLinus Torvalds /* TCP_ESTABLISHED */ TCP_FIN_WAIT1 | TCP_ACTION_FIN, 18321da177e4SLinus Torvalds /* TCP_SYN_SENT */ TCP_CLOSE, 18331da177e4SLinus Torvalds /* TCP_SYN_RECV */ TCP_FIN_WAIT1 | TCP_ACTION_FIN, 18341da177e4SLinus Torvalds /* TCP_FIN_WAIT1 */ TCP_FIN_WAIT1, 18351da177e4SLinus Torvalds /* TCP_FIN_WAIT2 */ TCP_FIN_WAIT2, 18361da177e4SLinus Torvalds /* TCP_TIME_WAIT */ TCP_CLOSE, 18371da177e4SLinus Torvalds /* TCP_CLOSE */ TCP_CLOSE, 18381da177e4SLinus Torvalds /* TCP_CLOSE_WAIT */ TCP_LAST_ACK | TCP_ACTION_FIN, 18391da177e4SLinus Torvalds /* TCP_LAST_ACK */ TCP_LAST_ACK, 18401da177e4SLinus Torvalds /* TCP_LISTEN */ TCP_CLOSE, 18411da177e4SLinus Torvalds /* TCP_CLOSING */ TCP_CLOSING, 18421da177e4SLinus Torvalds }; 18431da177e4SLinus Torvalds 18441da177e4SLinus Torvalds static int tcp_close_state(struct sock *sk) 18451da177e4SLinus Torvalds { 18461da177e4SLinus Torvalds int next = (int)new_state[sk->sk_state]; 18471da177e4SLinus Torvalds int ns = next & TCP_STATE_MASK; 18481da177e4SLinus Torvalds 18491da177e4SLinus Torvalds tcp_set_state(sk, ns); 18501da177e4SLinus Torvalds 18511da177e4SLinus Torvalds return next & TCP_ACTION_FIN; 18521da177e4SLinus Torvalds } 18531da177e4SLinus Torvalds 18541da177e4SLinus Torvalds /* 18551da177e4SLinus Torvalds * Shutdown the sending side of a connection. Much like close except 18561f29b058SSatoru SATOH * that we don't receive shut down or sock_set_flag(sk, SOCK_DEAD). 18571da177e4SLinus Torvalds */ 18581da177e4SLinus Torvalds 18591da177e4SLinus Torvalds void tcp_shutdown(struct sock *sk, int how) 18601da177e4SLinus Torvalds { 18611da177e4SLinus Torvalds /* We need to grab some memory, and put together a FIN, 18621da177e4SLinus Torvalds * and then put it into the queue to be sent. 18631da177e4SLinus Torvalds * Tim MacKenzie(tym@dibbler.cs.monash.edu.au) 4 Dec '92. 18641da177e4SLinus Torvalds */ 18651da177e4SLinus Torvalds if (!(how & SEND_SHUTDOWN)) 18661da177e4SLinus Torvalds return; 18671da177e4SLinus Torvalds 18681da177e4SLinus Torvalds /* If we've already sent a FIN, or it's a closed state, skip this. */ 18691da177e4SLinus Torvalds if ((1 << sk->sk_state) & 18701da177e4SLinus Torvalds (TCPF_ESTABLISHED | TCPF_SYN_SENT | 18711da177e4SLinus Torvalds TCPF_SYN_RECV | TCPF_CLOSE_WAIT)) { 18721da177e4SLinus Torvalds /* Clear out any half completed packets. FIN if needed. */ 18731da177e4SLinus Torvalds if (tcp_close_state(sk)) 18741da177e4SLinus Torvalds tcp_send_fin(sk); 18751da177e4SLinus Torvalds } 18761da177e4SLinus Torvalds } 18774bc2f18bSEric Dumazet EXPORT_SYMBOL(tcp_shutdown); 18781da177e4SLinus Torvalds 18791da177e4SLinus Torvalds void tcp_close(struct sock *sk, long timeout) 18801da177e4SLinus Torvalds { 18811da177e4SLinus Torvalds struct sk_buff *skb; 18821da177e4SLinus Torvalds int data_was_unread = 0; 188375c2d907SHerbert Xu int state; 18841da177e4SLinus Torvalds 18851da177e4SLinus Torvalds lock_sock(sk); 18861da177e4SLinus Torvalds sk->sk_shutdown = SHUTDOWN_MASK; 18871da177e4SLinus Torvalds 18881da177e4SLinus Torvalds if (sk->sk_state == TCP_LISTEN) { 18891da177e4SLinus Torvalds tcp_set_state(sk, TCP_CLOSE); 18901da177e4SLinus Torvalds 18911da177e4SLinus Torvalds /* Special case. */ 18920a5578cfSArnaldo Carvalho de Melo inet_csk_listen_stop(sk); 18931da177e4SLinus Torvalds 18941da177e4SLinus Torvalds goto adjudge_to_death; 18951da177e4SLinus Torvalds } 18961da177e4SLinus Torvalds 18971da177e4SLinus Torvalds /* We need to flush the recv. buffs. We do this only on the 18981da177e4SLinus Torvalds * descriptor close, not protocol-sourced closes, because the 18991da177e4SLinus Torvalds * reader process may not have drained the data yet! 19001da177e4SLinus Torvalds */ 19011da177e4SLinus Torvalds while ((skb = __skb_dequeue(&sk->sk_receive_queue)) != NULL) { 19021da177e4SLinus Torvalds u32 len = TCP_SKB_CB(skb)->end_seq - TCP_SKB_CB(skb)->seq - 1903aa8223c7SArnaldo Carvalho de Melo tcp_hdr(skb)->fin; 19041da177e4SLinus Torvalds data_was_unread += len; 19051da177e4SLinus Torvalds __kfree_skb(skb); 19061da177e4SLinus Torvalds } 19071da177e4SLinus Torvalds 19083ab224beSHideo Aoki sk_mem_reclaim(sk); 19091da177e4SLinus Torvalds 1910565b7b2dSKonstantin Khorenko /* If socket has been already reset (e.g. in tcp_reset()) - kill it. */ 1911565b7b2dSKonstantin Khorenko if (sk->sk_state == TCP_CLOSE) 1912565b7b2dSKonstantin Khorenko goto adjudge_to_death; 1913565b7b2dSKonstantin Khorenko 191465bb723cSGerrit Renker /* As outlined in RFC 2525, section 2.17, we send a RST here because 191565bb723cSGerrit Renker * data was lost. To witness the awful effects of the old behavior of 191665bb723cSGerrit Renker * always doing a FIN, run an older 2.1.x kernel or 2.0.x, start a bulk 191765bb723cSGerrit Renker * GET in an FTP client, suspend the process, wait for the client to 191865bb723cSGerrit Renker * advertise a zero window, then kill -9 the FTP client, wheee... 191965bb723cSGerrit Renker * Note: timeout is always zero in such a case. 19201da177e4SLinus Torvalds */ 19211da177e4SLinus Torvalds if (data_was_unread) { 19221da177e4SLinus Torvalds /* Unread data was tossed, zap the connection. */ 19236f67c817SPavel Emelyanov NET_INC_STATS_USER(sock_net(sk), LINUX_MIB_TCPABORTONCLOSE); 19241da177e4SLinus Torvalds tcp_set_state(sk, TCP_CLOSE); 1925aa133076SWu Fengguang tcp_send_active_reset(sk, sk->sk_allocation); 19261da177e4SLinus Torvalds } else if (sock_flag(sk, SOCK_LINGER) && !sk->sk_lingertime) { 19271da177e4SLinus Torvalds /* Check zero linger _after_ checking for unread data. */ 19281da177e4SLinus Torvalds sk->sk_prot->disconnect(sk, 0); 19296f67c817SPavel Emelyanov NET_INC_STATS_USER(sock_net(sk), LINUX_MIB_TCPABORTONDATA); 19301da177e4SLinus Torvalds } else if (tcp_close_state(sk)) { 19311da177e4SLinus Torvalds /* We FIN if the application ate all the data before 19321da177e4SLinus Torvalds * zapping the connection. 19331da177e4SLinus Torvalds */ 19341da177e4SLinus Torvalds 19351da177e4SLinus Torvalds /* RED-PEN. Formally speaking, we have broken TCP state 19361da177e4SLinus Torvalds * machine. State transitions: 19371da177e4SLinus Torvalds * 19381da177e4SLinus Torvalds * TCP_ESTABLISHED -> TCP_FIN_WAIT1 19391da177e4SLinus Torvalds * TCP_SYN_RECV -> TCP_FIN_WAIT1 (forget it, it's impossible) 19401da177e4SLinus Torvalds * TCP_CLOSE_WAIT -> TCP_LAST_ACK 19411da177e4SLinus Torvalds * 19421da177e4SLinus Torvalds * are legal only when FIN has been sent (i.e. in window), 19431da177e4SLinus Torvalds * rather than queued out of window. Purists blame. 19441da177e4SLinus Torvalds * 19451da177e4SLinus Torvalds * F.e. "RFC state" is ESTABLISHED, 19461da177e4SLinus Torvalds * if Linux state is FIN-WAIT-1, but FIN is still not sent. 19471da177e4SLinus Torvalds * 19481da177e4SLinus Torvalds * The visible declinations are that sometimes 19491da177e4SLinus Torvalds * we enter time-wait state, when it is not required really 19501da177e4SLinus Torvalds * (harmless), do not send active resets, when they are 19511da177e4SLinus Torvalds * required by specs (TCP_ESTABLISHED, TCP_CLOSE_WAIT, when 19521da177e4SLinus Torvalds * they look as CLOSING or LAST_ACK for Linux) 19531da177e4SLinus Torvalds * Probably, I missed some more holelets. 19541da177e4SLinus Torvalds * --ANK 19551da177e4SLinus Torvalds */ 19561da177e4SLinus Torvalds tcp_send_fin(sk); 19571da177e4SLinus Torvalds } 19581da177e4SLinus Torvalds 19591da177e4SLinus Torvalds sk_stream_wait_close(sk, timeout); 19601da177e4SLinus Torvalds 19611da177e4SLinus Torvalds adjudge_to_death: 196275c2d907SHerbert Xu state = sk->sk_state; 196375c2d907SHerbert Xu sock_hold(sk); 196475c2d907SHerbert Xu sock_orphan(sk); 196575c2d907SHerbert Xu 19661da177e4SLinus Torvalds /* It is the last release_sock in its life. It will remove backlog. */ 19671da177e4SLinus Torvalds release_sock(sk); 19681da177e4SLinus Torvalds 19691da177e4SLinus Torvalds 19701da177e4SLinus Torvalds /* Now socket is owned by kernel and we acquire BH lock 19711da177e4SLinus Torvalds to finish close. No need to check for user refs. 19721da177e4SLinus Torvalds */ 19731da177e4SLinus Torvalds local_bh_disable(); 19741da177e4SLinus Torvalds bh_lock_sock(sk); 1975547b792cSIlpo Järvinen WARN_ON(sock_owned_by_user(sk)); 19761da177e4SLinus Torvalds 1977eb4dea58SHerbert Xu percpu_counter_inc(sk->sk_prot->orphan_count); 1978eb4dea58SHerbert Xu 197975c2d907SHerbert Xu /* Have we already been destroyed by a softirq or backlog? */ 198075c2d907SHerbert Xu if (state != TCP_CLOSE && sk->sk_state == TCP_CLOSE) 198175c2d907SHerbert Xu goto out; 19821da177e4SLinus Torvalds 19831da177e4SLinus Torvalds /* This is a (useful) BSD violating of the RFC. There is a 19841da177e4SLinus Torvalds * problem with TCP as specified in that the other end could 19851da177e4SLinus Torvalds * keep a socket open forever with no application left this end. 19861da177e4SLinus Torvalds * We use a 3 minute timeout (about the same as BSD) then kill 19871da177e4SLinus Torvalds * our end. If they send after that then tough - BUT: long enough 19881da177e4SLinus Torvalds * that we won't make the old 4*rto = almost no time - whoops 19891da177e4SLinus Torvalds * reset mistake. 19901da177e4SLinus Torvalds * 19911da177e4SLinus Torvalds * Nope, it was not mistake. It is really desired behaviour 19921da177e4SLinus Torvalds * f.e. on http servers, when such sockets are useless, but 19931da177e4SLinus Torvalds * consume significant resources. Let's do it with special 19941da177e4SLinus Torvalds * linger2 option. --ANK 19951da177e4SLinus Torvalds */ 19961da177e4SLinus Torvalds 19971da177e4SLinus Torvalds if (sk->sk_state == TCP_FIN_WAIT2) { 19981da177e4SLinus Torvalds struct tcp_sock *tp = tcp_sk(sk); 19991da177e4SLinus Torvalds if (tp->linger2 < 0) { 20001da177e4SLinus Torvalds tcp_set_state(sk, TCP_CLOSE); 20011da177e4SLinus Torvalds tcp_send_active_reset(sk, GFP_ATOMIC); 2002de0744afSPavel Emelyanov NET_INC_STATS_BH(sock_net(sk), 2003de0744afSPavel Emelyanov LINUX_MIB_TCPABORTONLINGER); 20041da177e4SLinus Torvalds } else { 2005463c84b9SArnaldo Carvalho de Melo const int tmo = tcp_fin_time(sk); 20061da177e4SLinus Torvalds 20071da177e4SLinus Torvalds if (tmo > TCP_TIMEWAIT_LEN) { 200852499afeSDavid S. Miller inet_csk_reset_keepalive_timer(sk, 200952499afeSDavid S. Miller tmo - TCP_TIMEWAIT_LEN); 20101da177e4SLinus Torvalds } else { 20111da177e4SLinus Torvalds tcp_time_wait(sk, TCP_FIN_WAIT2, tmo); 20121da177e4SLinus Torvalds goto out; 20131da177e4SLinus Torvalds } 20141da177e4SLinus Torvalds } 20151da177e4SLinus Torvalds } 20161da177e4SLinus Torvalds if (sk->sk_state != TCP_CLOSE) { 20173ab224beSHideo Aoki sk_mem_reclaim(sk); 2018ad1af0feSDavid S. Miller if (tcp_too_many_orphans(sk, 0)) { 20191da177e4SLinus Torvalds if (net_ratelimit()) 20201da177e4SLinus Torvalds printk(KERN_INFO "TCP: too many of orphaned " 20211da177e4SLinus Torvalds "sockets\n"); 20221da177e4SLinus Torvalds tcp_set_state(sk, TCP_CLOSE); 20231da177e4SLinus Torvalds tcp_send_active_reset(sk, GFP_ATOMIC); 2024de0744afSPavel Emelyanov NET_INC_STATS_BH(sock_net(sk), 2025de0744afSPavel Emelyanov LINUX_MIB_TCPABORTONMEMORY); 20261da177e4SLinus Torvalds } 20271da177e4SLinus Torvalds } 20281da177e4SLinus Torvalds 20291da177e4SLinus Torvalds if (sk->sk_state == TCP_CLOSE) 20300a5578cfSArnaldo Carvalho de Melo inet_csk_destroy_sock(sk); 20311da177e4SLinus Torvalds /* Otherwise, socket is reprieved until protocol close. */ 20321da177e4SLinus Torvalds 20331da177e4SLinus Torvalds out: 20341da177e4SLinus Torvalds bh_unlock_sock(sk); 20351da177e4SLinus Torvalds local_bh_enable(); 20361da177e4SLinus Torvalds sock_put(sk); 20371da177e4SLinus Torvalds } 20384bc2f18bSEric Dumazet EXPORT_SYMBOL(tcp_close); 20391da177e4SLinus Torvalds 20401da177e4SLinus Torvalds /* These states need RST on ABORT according to RFC793 */ 20411da177e4SLinus Torvalds 20421da177e4SLinus Torvalds static inline int tcp_need_reset(int state) 20431da177e4SLinus Torvalds { 20441da177e4SLinus Torvalds return (1 << state) & 20451da177e4SLinus Torvalds (TCPF_ESTABLISHED | TCPF_CLOSE_WAIT | TCPF_FIN_WAIT1 | 20461da177e4SLinus Torvalds TCPF_FIN_WAIT2 | TCPF_SYN_RECV); 20471da177e4SLinus Torvalds } 20481da177e4SLinus Torvalds 20491da177e4SLinus Torvalds int tcp_disconnect(struct sock *sk, int flags) 20501da177e4SLinus Torvalds { 20511da177e4SLinus Torvalds struct inet_sock *inet = inet_sk(sk); 2052463c84b9SArnaldo Carvalho de Melo struct inet_connection_sock *icsk = inet_csk(sk); 20531da177e4SLinus Torvalds struct tcp_sock *tp = tcp_sk(sk); 20541da177e4SLinus Torvalds int err = 0; 20551da177e4SLinus Torvalds int old_state = sk->sk_state; 20561da177e4SLinus Torvalds 20571da177e4SLinus Torvalds if (old_state != TCP_CLOSE) 20581da177e4SLinus Torvalds tcp_set_state(sk, TCP_CLOSE); 20591da177e4SLinus Torvalds 20601da177e4SLinus Torvalds /* ABORT function of RFC793 */ 20611da177e4SLinus Torvalds if (old_state == TCP_LISTEN) { 20620a5578cfSArnaldo Carvalho de Melo inet_csk_listen_stop(sk); 20631da177e4SLinus Torvalds } else if (tcp_need_reset(old_state) || 20641da177e4SLinus Torvalds (tp->snd_nxt != tp->write_seq && 20651da177e4SLinus Torvalds (1 << old_state) & (TCPF_CLOSING | TCPF_LAST_ACK))) { 2066caa20d9aSStephen Hemminger /* The last check adjusts for discrepancy of Linux wrt. RFC 20671da177e4SLinus Torvalds * states 20681da177e4SLinus Torvalds */ 20691da177e4SLinus Torvalds tcp_send_active_reset(sk, gfp_any()); 20701da177e4SLinus Torvalds sk->sk_err = ECONNRESET; 20711da177e4SLinus Torvalds } else if (old_state == TCP_SYN_SENT) 20721da177e4SLinus Torvalds sk->sk_err = ECONNRESET; 20731da177e4SLinus Torvalds 20741da177e4SLinus Torvalds tcp_clear_xmit_timers(sk); 20751da177e4SLinus Torvalds __skb_queue_purge(&sk->sk_receive_queue); 2076fe067e8aSDavid S. Miller tcp_write_queue_purge(sk); 20771da177e4SLinus Torvalds __skb_queue_purge(&tp->out_of_order_queue); 20781a2449a8SChris Leech #ifdef CONFIG_NET_DMA 20791a2449a8SChris Leech __skb_queue_purge(&sk->sk_async_wait_queue); 20801a2449a8SChris Leech #endif 20811da177e4SLinus Torvalds 2082c720c7e8SEric Dumazet inet->inet_dport = 0; 20831da177e4SLinus Torvalds 20841da177e4SLinus Torvalds if (!(sk->sk_userlocks & SOCK_BINDADDR_LOCK)) 20851da177e4SLinus Torvalds inet_reset_saddr(sk); 20861da177e4SLinus Torvalds 20871da177e4SLinus Torvalds sk->sk_shutdown = 0; 20881da177e4SLinus Torvalds sock_reset_flag(sk, SOCK_DONE); 20891da177e4SLinus Torvalds tp->srtt = 0; 20901da177e4SLinus Torvalds if ((tp->write_seq += tp->max_window + 2) == 0) 20911da177e4SLinus Torvalds tp->write_seq = 1; 2092463c84b9SArnaldo Carvalho de Melo icsk->icsk_backoff = 0; 20931da177e4SLinus Torvalds tp->snd_cwnd = 2; 20946687e988SArnaldo Carvalho de Melo icsk->icsk_probes_out = 0; 20951da177e4SLinus Torvalds tp->packets_out = 0; 20960b6a05c1SIlpo Järvinen tp->snd_ssthresh = TCP_INFINITE_SSTHRESH; 20971da177e4SLinus Torvalds tp->snd_cwnd_cnt = 0; 20989772efb9SStephen Hemminger tp->bytes_acked = 0; 20991fdf475aSEric Dumazet tp->window_clamp = 0; 21006687e988SArnaldo Carvalho de Melo tcp_set_ca_state(sk, TCP_CA_Open); 21011da177e4SLinus Torvalds tcp_clear_retrans(tp); 2102463c84b9SArnaldo Carvalho de Melo inet_csk_delack_init(sk); 2103fe067e8aSDavid S. Miller tcp_init_send_head(sk); 2104b40b4f79SSrinivas Aji memset(&tp->rx_opt, 0, sizeof(tp->rx_opt)); 21051da177e4SLinus Torvalds __sk_dst_reset(sk); 21061da177e4SLinus Torvalds 2107c720c7e8SEric Dumazet WARN_ON(inet->inet_num && !icsk->icsk_bind_hash); 21081da177e4SLinus Torvalds 21091da177e4SLinus Torvalds sk->sk_error_report(sk); 21101da177e4SLinus Torvalds return err; 21111da177e4SLinus Torvalds } 21124bc2f18bSEric Dumazet EXPORT_SYMBOL(tcp_disconnect); 21131da177e4SLinus Torvalds 21141da177e4SLinus Torvalds /* 21151da177e4SLinus Torvalds * Socket option code for TCP. 21161da177e4SLinus Torvalds */ 21173fdadf7dSDmitry Mishin static int do_tcp_setsockopt(struct sock *sk, int level, 2118b7058842SDavid S. Miller int optname, char __user *optval, unsigned int optlen) 21191da177e4SLinus Torvalds { 21201da177e4SLinus Torvalds struct tcp_sock *tp = tcp_sk(sk); 2121463c84b9SArnaldo Carvalho de Melo struct inet_connection_sock *icsk = inet_csk(sk); 21221da177e4SLinus Torvalds int val; 21231da177e4SLinus Torvalds int err = 0; 21241da177e4SLinus Torvalds 2125e56fb50fSWilliam Allen Simpson /* These are data/string values, all the others are ints */ 2126e56fb50fSWilliam Allen Simpson switch (optname) { 2127e56fb50fSWilliam Allen Simpson case TCP_CONGESTION: { 21285f8ef48dSStephen Hemminger char name[TCP_CA_NAME_MAX]; 21295f8ef48dSStephen Hemminger 21305f8ef48dSStephen Hemminger if (optlen < 1) 21315f8ef48dSStephen Hemminger return -EINVAL; 21325f8ef48dSStephen Hemminger 21335f8ef48dSStephen Hemminger val = strncpy_from_user(name, optval, 21344fdb78d3SAndrew Morton min_t(long, TCP_CA_NAME_MAX-1, optlen)); 21355f8ef48dSStephen Hemminger if (val < 0) 21365f8ef48dSStephen Hemminger return -EFAULT; 21375f8ef48dSStephen Hemminger name[val] = 0; 21385f8ef48dSStephen Hemminger 21395f8ef48dSStephen Hemminger lock_sock(sk); 21406687e988SArnaldo Carvalho de Melo err = tcp_set_congestion_control(sk, name); 21415f8ef48dSStephen Hemminger release_sock(sk); 21425f8ef48dSStephen Hemminger return err; 21435f8ef48dSStephen Hemminger } 2144e56fb50fSWilliam Allen Simpson case TCP_COOKIE_TRANSACTIONS: { 2145e56fb50fSWilliam Allen Simpson struct tcp_cookie_transactions ctd; 2146e56fb50fSWilliam Allen Simpson struct tcp_cookie_values *cvp = NULL; 2147e56fb50fSWilliam Allen Simpson 2148e56fb50fSWilliam Allen Simpson if (sizeof(ctd) > optlen) 2149e56fb50fSWilliam Allen Simpson return -EINVAL; 2150e56fb50fSWilliam Allen Simpson if (copy_from_user(&ctd, optval, sizeof(ctd))) 2151e56fb50fSWilliam Allen Simpson return -EFAULT; 2152e56fb50fSWilliam Allen Simpson 2153e56fb50fSWilliam Allen Simpson if (ctd.tcpct_used > sizeof(ctd.tcpct_value) || 2154e56fb50fSWilliam Allen Simpson ctd.tcpct_s_data_desired > TCP_MSS_DESIRED) 2155e56fb50fSWilliam Allen Simpson return -EINVAL; 2156e56fb50fSWilliam Allen Simpson 2157e56fb50fSWilliam Allen Simpson if (ctd.tcpct_cookie_desired == 0) { 2158e56fb50fSWilliam Allen Simpson /* default to global value */ 2159e56fb50fSWilliam Allen Simpson } else if ((0x1 & ctd.tcpct_cookie_desired) || 2160e56fb50fSWilliam Allen Simpson ctd.tcpct_cookie_desired > TCP_COOKIE_MAX || 2161e56fb50fSWilliam Allen Simpson ctd.tcpct_cookie_desired < TCP_COOKIE_MIN) { 2162e56fb50fSWilliam Allen Simpson return -EINVAL; 2163e56fb50fSWilliam Allen Simpson } 2164e56fb50fSWilliam Allen Simpson 2165e56fb50fSWilliam Allen Simpson if (TCP_COOKIE_OUT_NEVER & ctd.tcpct_flags) { 2166e56fb50fSWilliam Allen Simpson /* Supercedes all other values */ 2167e56fb50fSWilliam Allen Simpson lock_sock(sk); 2168e56fb50fSWilliam Allen Simpson if (tp->cookie_values != NULL) { 2169e56fb50fSWilliam Allen Simpson kref_put(&tp->cookie_values->kref, 2170e56fb50fSWilliam Allen Simpson tcp_cookie_values_release); 2171e56fb50fSWilliam Allen Simpson tp->cookie_values = NULL; 2172e56fb50fSWilliam Allen Simpson } 2173e56fb50fSWilliam Allen Simpson tp->rx_opt.cookie_in_always = 0; /* false */ 2174e56fb50fSWilliam Allen Simpson tp->rx_opt.cookie_out_never = 1; /* true */ 2175e56fb50fSWilliam Allen Simpson release_sock(sk); 2176e56fb50fSWilliam Allen Simpson return err; 2177e56fb50fSWilliam Allen Simpson } 2178e56fb50fSWilliam Allen Simpson 2179e56fb50fSWilliam Allen Simpson /* Allocate ancillary memory before locking. 2180e56fb50fSWilliam Allen Simpson */ 2181e56fb50fSWilliam Allen Simpson if (ctd.tcpct_used > 0 || 2182e56fb50fSWilliam Allen Simpson (tp->cookie_values == NULL && 2183e56fb50fSWilliam Allen Simpson (sysctl_tcp_cookie_size > 0 || 2184e56fb50fSWilliam Allen Simpson ctd.tcpct_cookie_desired > 0 || 2185e56fb50fSWilliam Allen Simpson ctd.tcpct_s_data_desired > 0))) { 2186e56fb50fSWilliam Allen Simpson cvp = kzalloc(sizeof(*cvp) + ctd.tcpct_used, 2187e56fb50fSWilliam Allen Simpson GFP_KERNEL); 2188e56fb50fSWilliam Allen Simpson if (cvp == NULL) 2189e56fb50fSWilliam Allen Simpson return -ENOMEM; 2190a3bdb549SDmitry Popov 2191a3bdb549SDmitry Popov kref_init(&cvp->kref); 2192e56fb50fSWilliam Allen Simpson } 2193e56fb50fSWilliam Allen Simpson lock_sock(sk); 2194e56fb50fSWilliam Allen Simpson tp->rx_opt.cookie_in_always = 2195e56fb50fSWilliam Allen Simpson (TCP_COOKIE_IN_ALWAYS & ctd.tcpct_flags); 2196e56fb50fSWilliam Allen Simpson tp->rx_opt.cookie_out_never = 0; /* false */ 2197e56fb50fSWilliam Allen Simpson 2198e56fb50fSWilliam Allen Simpson if (tp->cookie_values != NULL) { 2199e56fb50fSWilliam Allen Simpson if (cvp != NULL) { 2200e56fb50fSWilliam Allen Simpson /* Changed values are recorded by a changed 2201e56fb50fSWilliam Allen Simpson * pointer, ensuring the cookie will differ, 2202e56fb50fSWilliam Allen Simpson * without separately hashing each value later. 2203e56fb50fSWilliam Allen Simpson */ 2204e56fb50fSWilliam Allen Simpson kref_put(&tp->cookie_values->kref, 2205e56fb50fSWilliam Allen Simpson tcp_cookie_values_release); 2206e56fb50fSWilliam Allen Simpson } else { 2207e56fb50fSWilliam Allen Simpson cvp = tp->cookie_values; 2208e56fb50fSWilliam Allen Simpson } 2209e56fb50fSWilliam Allen Simpson } 2210a3bdb549SDmitry Popov 2211e56fb50fSWilliam Allen Simpson if (cvp != NULL) { 2212e56fb50fSWilliam Allen Simpson cvp->cookie_desired = ctd.tcpct_cookie_desired; 2213e56fb50fSWilliam Allen Simpson 2214e56fb50fSWilliam Allen Simpson if (ctd.tcpct_used > 0) { 2215e56fb50fSWilliam Allen Simpson memcpy(cvp->s_data_payload, ctd.tcpct_value, 2216e56fb50fSWilliam Allen Simpson ctd.tcpct_used); 2217e56fb50fSWilliam Allen Simpson cvp->s_data_desired = ctd.tcpct_used; 2218e56fb50fSWilliam Allen Simpson cvp->s_data_constant = 1; /* true */ 2219e56fb50fSWilliam Allen Simpson } else { 2220e56fb50fSWilliam Allen Simpson /* No constant payload data. */ 2221e56fb50fSWilliam Allen Simpson cvp->s_data_desired = ctd.tcpct_s_data_desired; 2222e56fb50fSWilliam Allen Simpson cvp->s_data_constant = 0; /* false */ 2223e56fb50fSWilliam Allen Simpson } 2224a3bdb549SDmitry Popov 2225a3bdb549SDmitry Popov tp->cookie_values = cvp; 2226e56fb50fSWilliam Allen Simpson } 2227e56fb50fSWilliam Allen Simpson release_sock(sk); 2228e56fb50fSWilliam Allen Simpson return err; 2229e56fb50fSWilliam Allen Simpson } 2230e56fb50fSWilliam Allen Simpson default: 2231e56fb50fSWilliam Allen Simpson /* fallthru */ 2232e56fb50fSWilliam Allen Simpson break; 2233ccbd6a5aSJoe Perches } 22345f8ef48dSStephen Hemminger 22351da177e4SLinus Torvalds if (optlen < sizeof(int)) 22361da177e4SLinus Torvalds return -EINVAL; 22371da177e4SLinus Torvalds 22381da177e4SLinus Torvalds if (get_user(val, (int __user *)optval)) 22391da177e4SLinus Torvalds return -EFAULT; 22401da177e4SLinus Torvalds 22411da177e4SLinus Torvalds lock_sock(sk); 22421da177e4SLinus Torvalds 22431da177e4SLinus Torvalds switch (optname) { 22441da177e4SLinus Torvalds case TCP_MAXSEG: 22451da177e4SLinus Torvalds /* Values greater than interface MTU won't take effect. However 22461da177e4SLinus Torvalds * at the point when this call is done we typically don't yet 22471da177e4SLinus Torvalds * know which interface is going to be used */ 2248c39508d6SDavid S. Miller if (val < TCP_MIN_MSS || val > MAX_TCP_WINDOW) { 22491da177e4SLinus Torvalds err = -EINVAL; 22501da177e4SLinus Torvalds break; 22511da177e4SLinus Torvalds } 22521da177e4SLinus Torvalds tp->rx_opt.user_mss = val; 22531da177e4SLinus Torvalds break; 22541da177e4SLinus Torvalds 22551da177e4SLinus Torvalds case TCP_NODELAY: 22561da177e4SLinus Torvalds if (val) { 22571da177e4SLinus Torvalds /* TCP_NODELAY is weaker than TCP_CORK, so that 22581da177e4SLinus Torvalds * this option on corked socket is remembered, but 22591da177e4SLinus Torvalds * it is not activated until cork is cleared. 22601da177e4SLinus Torvalds * 22611da177e4SLinus Torvalds * However, when TCP_NODELAY is set we make 22621da177e4SLinus Torvalds * an explicit push, which overrides even TCP_CORK 22631da177e4SLinus Torvalds * for currently queued segments. 22641da177e4SLinus Torvalds */ 22651da177e4SLinus Torvalds tp->nonagle |= TCP_NAGLE_OFF|TCP_NAGLE_PUSH; 22669e412ba7SIlpo Järvinen tcp_push_pending_frames(sk); 22671da177e4SLinus Torvalds } else { 22681da177e4SLinus Torvalds tp->nonagle &= ~TCP_NAGLE_OFF; 22691da177e4SLinus Torvalds } 22701da177e4SLinus Torvalds break; 22711da177e4SLinus Torvalds 227236e31b0aSAndreas Petlund case TCP_THIN_LINEAR_TIMEOUTS: 227336e31b0aSAndreas Petlund if (val < 0 || val > 1) 227436e31b0aSAndreas Petlund err = -EINVAL; 227536e31b0aSAndreas Petlund else 227636e31b0aSAndreas Petlund tp->thin_lto = val; 227736e31b0aSAndreas Petlund break; 227836e31b0aSAndreas Petlund 22797e380175SAndreas Petlund case TCP_THIN_DUPACK: 22807e380175SAndreas Petlund if (val < 0 || val > 1) 22817e380175SAndreas Petlund err = -EINVAL; 22827e380175SAndreas Petlund else 22837e380175SAndreas Petlund tp->thin_dupack = val; 22847e380175SAndreas Petlund break; 22857e380175SAndreas Petlund 22861da177e4SLinus Torvalds case TCP_CORK: 22871da177e4SLinus Torvalds /* When set indicates to always queue non-full frames. 22881da177e4SLinus Torvalds * Later the user clears this option and we transmit 22891da177e4SLinus Torvalds * any pending partial frames in the queue. This is 22901da177e4SLinus Torvalds * meant to be used alongside sendfile() to get properly 22911da177e4SLinus Torvalds * filled frames when the user (for example) must write 22921da177e4SLinus Torvalds * out headers with a write() call first and then use 22931da177e4SLinus Torvalds * sendfile to send out the data parts. 22941da177e4SLinus Torvalds * 22951da177e4SLinus Torvalds * TCP_CORK can be set together with TCP_NODELAY and it is 22961da177e4SLinus Torvalds * stronger than TCP_NODELAY. 22971da177e4SLinus Torvalds */ 22981da177e4SLinus Torvalds if (val) { 22991da177e4SLinus Torvalds tp->nonagle |= TCP_NAGLE_CORK; 23001da177e4SLinus Torvalds } else { 23011da177e4SLinus Torvalds tp->nonagle &= ~TCP_NAGLE_CORK; 23021da177e4SLinus Torvalds if (tp->nonagle&TCP_NAGLE_OFF) 23031da177e4SLinus Torvalds tp->nonagle |= TCP_NAGLE_PUSH; 23049e412ba7SIlpo Järvinen tcp_push_pending_frames(sk); 23051da177e4SLinus Torvalds } 23061da177e4SLinus Torvalds break; 23071da177e4SLinus Torvalds 23081da177e4SLinus Torvalds case TCP_KEEPIDLE: 23091da177e4SLinus Torvalds if (val < 1 || val > MAX_TCP_KEEPIDLE) 23101da177e4SLinus Torvalds err = -EINVAL; 23111da177e4SLinus Torvalds else { 23121da177e4SLinus Torvalds tp->keepalive_time = val * HZ; 23131da177e4SLinus Torvalds if (sock_flag(sk, SOCK_KEEPOPEN) && 23141da177e4SLinus Torvalds !((1 << sk->sk_state) & 23151da177e4SLinus Torvalds (TCPF_CLOSE | TCPF_LISTEN))) { 23166c37e5deSFlavio Leitner u32 elapsed = keepalive_time_elapsed(tp); 23171da177e4SLinus Torvalds if (tp->keepalive_time > elapsed) 23181da177e4SLinus Torvalds elapsed = tp->keepalive_time - elapsed; 23191da177e4SLinus Torvalds else 23201da177e4SLinus Torvalds elapsed = 0; 2321463c84b9SArnaldo Carvalho de Melo inet_csk_reset_keepalive_timer(sk, elapsed); 23221da177e4SLinus Torvalds } 23231da177e4SLinus Torvalds } 23241da177e4SLinus Torvalds break; 23251da177e4SLinus Torvalds case TCP_KEEPINTVL: 23261da177e4SLinus Torvalds if (val < 1 || val > MAX_TCP_KEEPINTVL) 23271da177e4SLinus Torvalds err = -EINVAL; 23281da177e4SLinus Torvalds else 23291da177e4SLinus Torvalds tp->keepalive_intvl = val * HZ; 23301da177e4SLinus Torvalds break; 23311da177e4SLinus Torvalds case TCP_KEEPCNT: 23321da177e4SLinus Torvalds if (val < 1 || val > MAX_TCP_KEEPCNT) 23331da177e4SLinus Torvalds err = -EINVAL; 23341da177e4SLinus Torvalds else 23351da177e4SLinus Torvalds tp->keepalive_probes = val; 23361da177e4SLinus Torvalds break; 23371da177e4SLinus Torvalds case TCP_SYNCNT: 23381da177e4SLinus Torvalds if (val < 1 || val > MAX_TCP_SYNCNT) 23391da177e4SLinus Torvalds err = -EINVAL; 23401da177e4SLinus Torvalds else 2341463c84b9SArnaldo Carvalho de Melo icsk->icsk_syn_retries = val; 23421da177e4SLinus Torvalds break; 23431da177e4SLinus Torvalds 23441da177e4SLinus Torvalds case TCP_LINGER2: 23451da177e4SLinus Torvalds if (val < 0) 23461da177e4SLinus Torvalds tp->linger2 = -1; 23471da177e4SLinus Torvalds else if (val > sysctl_tcp_fin_timeout / HZ) 23481da177e4SLinus Torvalds tp->linger2 = 0; 23491da177e4SLinus Torvalds else 23501da177e4SLinus Torvalds tp->linger2 = val * HZ; 23511da177e4SLinus Torvalds break; 23521da177e4SLinus Torvalds 23531da177e4SLinus Torvalds case TCP_DEFER_ACCEPT: 2354b103cf34SJulian Anastasov /* Translate value in seconds to number of retransmits */ 2355b103cf34SJulian Anastasov icsk->icsk_accept_queue.rskq_defer_accept = 2356b103cf34SJulian Anastasov secs_to_retrans(val, TCP_TIMEOUT_INIT / HZ, 2357b103cf34SJulian Anastasov TCP_RTO_MAX / HZ); 23581da177e4SLinus Torvalds break; 23591da177e4SLinus Torvalds 23601da177e4SLinus Torvalds case TCP_WINDOW_CLAMP: 23611da177e4SLinus Torvalds if (!val) { 23621da177e4SLinus Torvalds if (sk->sk_state != TCP_CLOSE) { 23631da177e4SLinus Torvalds err = -EINVAL; 23641da177e4SLinus Torvalds break; 23651da177e4SLinus Torvalds } 23661da177e4SLinus Torvalds tp->window_clamp = 0; 23671da177e4SLinus Torvalds } else 23681da177e4SLinus Torvalds tp->window_clamp = val < SOCK_MIN_RCVBUF / 2 ? 23691da177e4SLinus Torvalds SOCK_MIN_RCVBUF / 2 : val; 23701da177e4SLinus Torvalds break; 23711da177e4SLinus Torvalds 23721da177e4SLinus Torvalds case TCP_QUICKACK: 23731da177e4SLinus Torvalds if (!val) { 2374463c84b9SArnaldo Carvalho de Melo icsk->icsk_ack.pingpong = 1; 23751da177e4SLinus Torvalds } else { 2376463c84b9SArnaldo Carvalho de Melo icsk->icsk_ack.pingpong = 0; 23771da177e4SLinus Torvalds if ((1 << sk->sk_state) & 23781da177e4SLinus Torvalds (TCPF_ESTABLISHED | TCPF_CLOSE_WAIT) && 2379463c84b9SArnaldo Carvalho de Melo inet_csk_ack_scheduled(sk)) { 2380463c84b9SArnaldo Carvalho de Melo icsk->icsk_ack.pending |= ICSK_ACK_PUSHED; 23810e4b4992SChris Leech tcp_cleanup_rbuf(sk, 1); 23821da177e4SLinus Torvalds if (!(val & 1)) 2383463c84b9SArnaldo Carvalho de Melo icsk->icsk_ack.pingpong = 1; 23841da177e4SLinus Torvalds } 23851da177e4SLinus Torvalds } 23861da177e4SLinus Torvalds break; 23871da177e4SLinus Torvalds 2388cfb6eeb4SYOSHIFUJI Hideaki #ifdef CONFIG_TCP_MD5SIG 2389cfb6eeb4SYOSHIFUJI Hideaki case TCP_MD5SIG: 2390cfb6eeb4SYOSHIFUJI Hideaki /* Read the IP->Key mappings from userspace */ 2391cfb6eeb4SYOSHIFUJI Hideaki err = tp->af_specific->md5_parse(sk, optval, optlen); 2392cfb6eeb4SYOSHIFUJI Hideaki break; 2393cfb6eeb4SYOSHIFUJI Hideaki #endif 2394dca43c75SJerry Chu case TCP_USER_TIMEOUT: 2395dca43c75SJerry Chu /* Cap the max timeout in ms TCP will retry/retrans 2396dca43c75SJerry Chu * before giving up and aborting (ETIMEDOUT) a connection. 2397dca43c75SJerry Chu */ 2398dca43c75SJerry Chu icsk->icsk_user_timeout = msecs_to_jiffies(val); 2399dca43c75SJerry Chu break; 24001da177e4SLinus Torvalds default: 24011da177e4SLinus Torvalds err = -ENOPROTOOPT; 24021da177e4SLinus Torvalds break; 24033ff50b79SStephen Hemminger } 24043ff50b79SStephen Hemminger 24051da177e4SLinus Torvalds release_sock(sk); 24061da177e4SLinus Torvalds return err; 24071da177e4SLinus Torvalds } 24081da177e4SLinus Torvalds 24093fdadf7dSDmitry Mishin int tcp_setsockopt(struct sock *sk, int level, int optname, char __user *optval, 2410b7058842SDavid S. Miller unsigned int optlen) 24113fdadf7dSDmitry Mishin { 24123fdadf7dSDmitry Mishin struct inet_connection_sock *icsk = inet_csk(sk); 24133fdadf7dSDmitry Mishin 24143fdadf7dSDmitry Mishin if (level != SOL_TCP) 24153fdadf7dSDmitry Mishin return icsk->icsk_af_ops->setsockopt(sk, level, optname, 24163fdadf7dSDmitry Mishin optval, optlen); 24173fdadf7dSDmitry Mishin return do_tcp_setsockopt(sk, level, optname, optval, optlen); 24183fdadf7dSDmitry Mishin } 24194bc2f18bSEric Dumazet EXPORT_SYMBOL(tcp_setsockopt); 24203fdadf7dSDmitry Mishin 24213fdadf7dSDmitry Mishin #ifdef CONFIG_COMPAT 2422543d9cfeSArnaldo Carvalho de Melo int compat_tcp_setsockopt(struct sock *sk, int level, int optname, 2423b7058842SDavid S. Miller char __user *optval, unsigned int optlen) 24243fdadf7dSDmitry Mishin { 2425dec73ff0SArnaldo Carvalho de Melo if (level != SOL_TCP) 2426dec73ff0SArnaldo Carvalho de Melo return inet_csk_compat_setsockopt(sk, level, optname, 2427dec73ff0SArnaldo Carvalho de Melo optval, optlen); 24283fdadf7dSDmitry Mishin return do_tcp_setsockopt(sk, level, optname, optval, optlen); 24293fdadf7dSDmitry Mishin } 2430543d9cfeSArnaldo Carvalho de Melo EXPORT_SYMBOL(compat_tcp_setsockopt); 24313fdadf7dSDmitry Mishin #endif 24323fdadf7dSDmitry Mishin 24331da177e4SLinus Torvalds /* Return information about state of tcp endpoint in API format. */ 24341da177e4SLinus Torvalds void tcp_get_info(struct sock *sk, struct tcp_info *info) 24351da177e4SLinus Torvalds { 24361da177e4SLinus Torvalds struct tcp_sock *tp = tcp_sk(sk); 2437463c84b9SArnaldo Carvalho de Melo const struct inet_connection_sock *icsk = inet_csk(sk); 24381da177e4SLinus Torvalds u32 now = tcp_time_stamp; 24391da177e4SLinus Torvalds 24401da177e4SLinus Torvalds memset(info, 0, sizeof(*info)); 24411da177e4SLinus Torvalds 24421da177e4SLinus Torvalds info->tcpi_state = sk->sk_state; 24436687e988SArnaldo Carvalho de Melo info->tcpi_ca_state = icsk->icsk_ca_state; 2444463c84b9SArnaldo Carvalho de Melo info->tcpi_retransmits = icsk->icsk_retransmits; 24456687e988SArnaldo Carvalho de Melo info->tcpi_probes = icsk->icsk_probes_out; 2446463c84b9SArnaldo Carvalho de Melo info->tcpi_backoff = icsk->icsk_backoff; 24471da177e4SLinus Torvalds 24481da177e4SLinus Torvalds if (tp->rx_opt.tstamp_ok) 24491da177e4SLinus Torvalds info->tcpi_options |= TCPI_OPT_TIMESTAMPS; 2450e60402d0SIlpo Järvinen if (tcp_is_sack(tp)) 24511da177e4SLinus Torvalds info->tcpi_options |= TCPI_OPT_SACK; 24521da177e4SLinus Torvalds if (tp->rx_opt.wscale_ok) { 24531da177e4SLinus Torvalds info->tcpi_options |= TCPI_OPT_WSCALE; 24541da177e4SLinus Torvalds info->tcpi_snd_wscale = tp->rx_opt.snd_wscale; 24551da177e4SLinus Torvalds info->tcpi_rcv_wscale = tp->rx_opt.rcv_wscale; 24561da177e4SLinus Torvalds } 24571da177e4SLinus Torvalds 24581da177e4SLinus Torvalds if (tp->ecn_flags & TCP_ECN_OK) 24591da177e4SLinus Torvalds info->tcpi_options |= TCPI_OPT_ECN; 2460*b5c5693bSEric Dumazet if (tp->ecn_flags & TCP_ECN_SEEN) 2461*b5c5693bSEric Dumazet info->tcpi_options |= TCPI_OPT_ECN_SEEN; 24621da177e4SLinus Torvalds 2463463c84b9SArnaldo Carvalho de Melo info->tcpi_rto = jiffies_to_usecs(icsk->icsk_rto); 2464463c84b9SArnaldo Carvalho de Melo info->tcpi_ato = jiffies_to_usecs(icsk->icsk_ack.ato); 2465c1b4a7e6SDavid S. Miller info->tcpi_snd_mss = tp->mss_cache; 2466463c84b9SArnaldo Carvalho de Melo info->tcpi_rcv_mss = icsk->icsk_ack.rcv_mss; 24671da177e4SLinus Torvalds 24685ee3afbaSRick Jones if (sk->sk_state == TCP_LISTEN) { 24695ee3afbaSRick Jones info->tcpi_unacked = sk->sk_ack_backlog; 24705ee3afbaSRick Jones info->tcpi_sacked = sk->sk_max_ack_backlog; 24715ee3afbaSRick Jones } else { 24721da177e4SLinus Torvalds info->tcpi_unacked = tp->packets_out; 24731da177e4SLinus Torvalds info->tcpi_sacked = tp->sacked_out; 24745ee3afbaSRick Jones } 24751da177e4SLinus Torvalds info->tcpi_lost = tp->lost_out; 24761da177e4SLinus Torvalds info->tcpi_retrans = tp->retrans_out; 24771da177e4SLinus Torvalds info->tcpi_fackets = tp->fackets_out; 24781da177e4SLinus Torvalds 24791da177e4SLinus Torvalds info->tcpi_last_data_sent = jiffies_to_msecs(now - tp->lsndtime); 2480463c84b9SArnaldo Carvalho de Melo info->tcpi_last_data_recv = jiffies_to_msecs(now - icsk->icsk_ack.lrcvtime); 24811da177e4SLinus Torvalds info->tcpi_last_ack_recv = jiffies_to_msecs(now - tp->rcv_tstamp); 24821da177e4SLinus Torvalds 2483d83d8461SArnaldo Carvalho de Melo info->tcpi_pmtu = icsk->icsk_pmtu_cookie; 24841da177e4SLinus Torvalds info->tcpi_rcv_ssthresh = tp->rcv_ssthresh; 24851da177e4SLinus Torvalds info->tcpi_rtt = jiffies_to_usecs(tp->srtt)>>3; 24861da177e4SLinus Torvalds info->tcpi_rttvar = jiffies_to_usecs(tp->mdev)>>2; 24871da177e4SLinus Torvalds info->tcpi_snd_ssthresh = tp->snd_ssthresh; 24881da177e4SLinus Torvalds info->tcpi_snd_cwnd = tp->snd_cwnd; 24891da177e4SLinus Torvalds info->tcpi_advmss = tp->advmss; 24901da177e4SLinus Torvalds info->tcpi_reordering = tp->reordering; 24911da177e4SLinus Torvalds 24921da177e4SLinus Torvalds info->tcpi_rcv_rtt = jiffies_to_usecs(tp->rcv_rtt_est.rtt)>>3; 24931da177e4SLinus Torvalds info->tcpi_rcv_space = tp->rcvq_space.space; 24941da177e4SLinus Torvalds 24951da177e4SLinus Torvalds info->tcpi_total_retrans = tp->total_retrans; 24961da177e4SLinus Torvalds } 24971da177e4SLinus Torvalds EXPORT_SYMBOL_GPL(tcp_get_info); 24981da177e4SLinus Torvalds 24993fdadf7dSDmitry Mishin static int do_tcp_getsockopt(struct sock *sk, int level, 25003fdadf7dSDmitry Mishin int optname, char __user *optval, int __user *optlen) 25011da177e4SLinus Torvalds { 2502295f7324SArnaldo Carvalho de Melo struct inet_connection_sock *icsk = inet_csk(sk); 25031da177e4SLinus Torvalds struct tcp_sock *tp = tcp_sk(sk); 25041da177e4SLinus Torvalds int val, len; 25051da177e4SLinus Torvalds 25061da177e4SLinus Torvalds if (get_user(len, optlen)) 25071da177e4SLinus Torvalds return -EFAULT; 25081da177e4SLinus Torvalds 25091da177e4SLinus Torvalds len = min_t(unsigned int, len, sizeof(int)); 25101da177e4SLinus Torvalds 25111da177e4SLinus Torvalds if (len < 0) 25121da177e4SLinus Torvalds return -EINVAL; 25131da177e4SLinus Torvalds 25141da177e4SLinus Torvalds switch (optname) { 25151da177e4SLinus Torvalds case TCP_MAXSEG: 2516c1b4a7e6SDavid S. Miller val = tp->mss_cache; 25171da177e4SLinus Torvalds if (!val && ((1 << sk->sk_state) & (TCPF_CLOSE | TCPF_LISTEN))) 25181da177e4SLinus Torvalds val = tp->rx_opt.user_mss; 25191da177e4SLinus Torvalds break; 25201da177e4SLinus Torvalds case TCP_NODELAY: 25211da177e4SLinus Torvalds val = !!(tp->nonagle&TCP_NAGLE_OFF); 25221da177e4SLinus Torvalds break; 25231da177e4SLinus Torvalds case TCP_CORK: 25241da177e4SLinus Torvalds val = !!(tp->nonagle&TCP_NAGLE_CORK); 25251da177e4SLinus Torvalds break; 25261da177e4SLinus Torvalds case TCP_KEEPIDLE: 2527df19a626SEric Dumazet val = keepalive_time_when(tp) / HZ; 25281da177e4SLinus Torvalds break; 25291da177e4SLinus Torvalds case TCP_KEEPINTVL: 2530df19a626SEric Dumazet val = keepalive_intvl_when(tp) / HZ; 25311da177e4SLinus Torvalds break; 25321da177e4SLinus Torvalds case TCP_KEEPCNT: 2533df19a626SEric Dumazet val = keepalive_probes(tp); 25341da177e4SLinus Torvalds break; 25351da177e4SLinus Torvalds case TCP_SYNCNT: 2536295f7324SArnaldo Carvalho de Melo val = icsk->icsk_syn_retries ? : sysctl_tcp_syn_retries; 25371da177e4SLinus Torvalds break; 25381da177e4SLinus Torvalds case TCP_LINGER2: 25391da177e4SLinus Torvalds val = tp->linger2; 25401da177e4SLinus Torvalds if (val >= 0) 25411da177e4SLinus Torvalds val = (val ? : sysctl_tcp_fin_timeout) / HZ; 25421da177e4SLinus Torvalds break; 25431da177e4SLinus Torvalds case TCP_DEFER_ACCEPT: 2544b103cf34SJulian Anastasov val = retrans_to_secs(icsk->icsk_accept_queue.rskq_defer_accept, 2545b103cf34SJulian Anastasov TCP_TIMEOUT_INIT / HZ, TCP_RTO_MAX / HZ); 25461da177e4SLinus Torvalds break; 25471da177e4SLinus Torvalds case TCP_WINDOW_CLAMP: 25481da177e4SLinus Torvalds val = tp->window_clamp; 25491da177e4SLinus Torvalds break; 25501da177e4SLinus Torvalds case TCP_INFO: { 25511da177e4SLinus Torvalds struct tcp_info info; 25521da177e4SLinus Torvalds 25531da177e4SLinus Torvalds if (get_user(len, optlen)) 25541da177e4SLinus Torvalds return -EFAULT; 25551da177e4SLinus Torvalds 25561da177e4SLinus Torvalds tcp_get_info(sk, &info); 25571da177e4SLinus Torvalds 25581da177e4SLinus Torvalds len = min_t(unsigned int, len, sizeof(info)); 25591da177e4SLinus Torvalds if (put_user(len, optlen)) 25601da177e4SLinus Torvalds return -EFAULT; 25611da177e4SLinus Torvalds if (copy_to_user(optval, &info, len)) 25621da177e4SLinus Torvalds return -EFAULT; 25631da177e4SLinus Torvalds return 0; 25641da177e4SLinus Torvalds } 25651da177e4SLinus Torvalds case TCP_QUICKACK: 2566295f7324SArnaldo Carvalho de Melo val = !icsk->icsk_ack.pingpong; 25671da177e4SLinus Torvalds break; 25685f8ef48dSStephen Hemminger 25695f8ef48dSStephen Hemminger case TCP_CONGESTION: 25705f8ef48dSStephen Hemminger if (get_user(len, optlen)) 25715f8ef48dSStephen Hemminger return -EFAULT; 25725f8ef48dSStephen Hemminger len = min_t(unsigned int, len, TCP_CA_NAME_MAX); 25735f8ef48dSStephen Hemminger if (put_user(len, optlen)) 25745f8ef48dSStephen Hemminger return -EFAULT; 25756687e988SArnaldo Carvalho de Melo if (copy_to_user(optval, icsk->icsk_ca_ops->name, len)) 25765f8ef48dSStephen Hemminger return -EFAULT; 25775f8ef48dSStephen Hemminger return 0; 2578e56fb50fSWilliam Allen Simpson 2579e56fb50fSWilliam Allen Simpson case TCP_COOKIE_TRANSACTIONS: { 2580e56fb50fSWilliam Allen Simpson struct tcp_cookie_transactions ctd; 2581e56fb50fSWilliam Allen Simpson struct tcp_cookie_values *cvp = tp->cookie_values; 2582e56fb50fSWilliam Allen Simpson 2583e56fb50fSWilliam Allen Simpson if (get_user(len, optlen)) 2584e56fb50fSWilliam Allen Simpson return -EFAULT; 2585e56fb50fSWilliam Allen Simpson if (len < sizeof(ctd)) 2586e56fb50fSWilliam Allen Simpson return -EINVAL; 2587e56fb50fSWilliam Allen Simpson 2588e56fb50fSWilliam Allen Simpson memset(&ctd, 0, sizeof(ctd)); 2589e56fb50fSWilliam Allen Simpson ctd.tcpct_flags = (tp->rx_opt.cookie_in_always ? 2590e56fb50fSWilliam Allen Simpson TCP_COOKIE_IN_ALWAYS : 0) 2591e56fb50fSWilliam Allen Simpson | (tp->rx_opt.cookie_out_never ? 2592e56fb50fSWilliam Allen Simpson TCP_COOKIE_OUT_NEVER : 0); 2593e56fb50fSWilliam Allen Simpson 2594e56fb50fSWilliam Allen Simpson if (cvp != NULL) { 2595e56fb50fSWilliam Allen Simpson ctd.tcpct_flags |= (cvp->s_data_in ? 2596e56fb50fSWilliam Allen Simpson TCP_S_DATA_IN : 0) 2597e56fb50fSWilliam Allen Simpson | (cvp->s_data_out ? 2598e56fb50fSWilliam Allen Simpson TCP_S_DATA_OUT : 0); 2599e56fb50fSWilliam Allen Simpson 2600e56fb50fSWilliam Allen Simpson ctd.tcpct_cookie_desired = cvp->cookie_desired; 2601e56fb50fSWilliam Allen Simpson ctd.tcpct_s_data_desired = cvp->s_data_desired; 2602e56fb50fSWilliam Allen Simpson 2603e56fb50fSWilliam Allen Simpson memcpy(&ctd.tcpct_value[0], &cvp->cookie_pair[0], 2604e56fb50fSWilliam Allen Simpson cvp->cookie_pair_size); 2605e56fb50fSWilliam Allen Simpson ctd.tcpct_used = cvp->cookie_pair_size; 2606e56fb50fSWilliam Allen Simpson } 2607e56fb50fSWilliam Allen Simpson 2608e56fb50fSWilliam Allen Simpson if (put_user(sizeof(ctd), optlen)) 2609e56fb50fSWilliam Allen Simpson return -EFAULT; 2610e56fb50fSWilliam Allen Simpson if (copy_to_user(optval, &ctd, sizeof(ctd))) 2611e56fb50fSWilliam Allen Simpson return -EFAULT; 2612e56fb50fSWilliam Allen Simpson return 0; 2613e56fb50fSWilliam Allen Simpson } 26143c0fef0bSJosh Hunt case TCP_THIN_LINEAR_TIMEOUTS: 26153c0fef0bSJosh Hunt val = tp->thin_lto; 26163c0fef0bSJosh Hunt break; 26173c0fef0bSJosh Hunt case TCP_THIN_DUPACK: 26183c0fef0bSJosh Hunt val = tp->thin_dupack; 26193c0fef0bSJosh Hunt break; 2620dca43c75SJerry Chu 2621dca43c75SJerry Chu case TCP_USER_TIMEOUT: 2622dca43c75SJerry Chu val = jiffies_to_msecs(icsk->icsk_user_timeout); 2623dca43c75SJerry Chu break; 26241da177e4SLinus Torvalds default: 26251da177e4SLinus Torvalds return -ENOPROTOOPT; 26263ff50b79SStephen Hemminger } 26271da177e4SLinus Torvalds 26281da177e4SLinus Torvalds if (put_user(len, optlen)) 26291da177e4SLinus Torvalds return -EFAULT; 26301da177e4SLinus Torvalds if (copy_to_user(optval, &val, len)) 26311da177e4SLinus Torvalds return -EFAULT; 26321da177e4SLinus Torvalds return 0; 26331da177e4SLinus Torvalds } 26341da177e4SLinus Torvalds 26353fdadf7dSDmitry Mishin int tcp_getsockopt(struct sock *sk, int level, int optname, char __user *optval, 26363fdadf7dSDmitry Mishin int __user *optlen) 26373fdadf7dSDmitry Mishin { 26383fdadf7dSDmitry Mishin struct inet_connection_sock *icsk = inet_csk(sk); 26393fdadf7dSDmitry Mishin 26403fdadf7dSDmitry Mishin if (level != SOL_TCP) 26413fdadf7dSDmitry Mishin return icsk->icsk_af_ops->getsockopt(sk, level, optname, 26423fdadf7dSDmitry Mishin optval, optlen); 26433fdadf7dSDmitry Mishin return do_tcp_getsockopt(sk, level, optname, optval, optlen); 26443fdadf7dSDmitry Mishin } 26454bc2f18bSEric Dumazet EXPORT_SYMBOL(tcp_getsockopt); 26463fdadf7dSDmitry Mishin 26473fdadf7dSDmitry Mishin #ifdef CONFIG_COMPAT 2648543d9cfeSArnaldo Carvalho de Melo int compat_tcp_getsockopt(struct sock *sk, int level, int optname, 2649543d9cfeSArnaldo Carvalho de Melo char __user *optval, int __user *optlen) 26503fdadf7dSDmitry Mishin { 2651dec73ff0SArnaldo Carvalho de Melo if (level != SOL_TCP) 2652dec73ff0SArnaldo Carvalho de Melo return inet_csk_compat_getsockopt(sk, level, optname, 2653dec73ff0SArnaldo Carvalho de Melo optval, optlen); 26543fdadf7dSDmitry Mishin return do_tcp_getsockopt(sk, level, optname, optval, optlen); 26553fdadf7dSDmitry Mishin } 2656543d9cfeSArnaldo Carvalho de Melo EXPORT_SYMBOL(compat_tcp_getsockopt); 26573fdadf7dSDmitry Mishin #endif 26581da177e4SLinus Torvalds 265904ed3e74SMichał Mirosław struct sk_buff *tcp_tso_segment(struct sk_buff *skb, u32 features) 2660f4c50d99SHerbert Xu { 2661f4c50d99SHerbert Xu struct sk_buff *segs = ERR_PTR(-EINVAL); 2662f4c50d99SHerbert Xu struct tcphdr *th; 2663f4c50d99SHerbert Xu unsigned thlen; 2664f4c50d99SHerbert Xu unsigned int seq; 2665d3bc23e7SAl Viro __be32 delta; 2666f4c50d99SHerbert Xu unsigned int oldlen; 26674e704ee3SHerbert Xu unsigned int mss; 2668f4c50d99SHerbert Xu 2669f4c50d99SHerbert Xu if (!pskb_may_pull(skb, sizeof(*th))) 2670f4c50d99SHerbert Xu goto out; 2671f4c50d99SHerbert Xu 2672aa8223c7SArnaldo Carvalho de Melo th = tcp_hdr(skb); 2673f4c50d99SHerbert Xu thlen = th->doff * 4; 2674f4c50d99SHerbert Xu if (thlen < sizeof(*th)) 2675f4c50d99SHerbert Xu goto out; 2676f4c50d99SHerbert Xu 2677f4c50d99SHerbert Xu if (!pskb_may_pull(skb, thlen)) 2678f4c50d99SHerbert Xu goto out; 2679f4c50d99SHerbert Xu 26800718bcc0SHerbert Xu oldlen = (u16)~skb->len; 2681f4c50d99SHerbert Xu __skb_pull(skb, thlen); 2682f4c50d99SHerbert Xu 26834e704ee3SHerbert Xu mss = skb_shinfo(skb)->gso_size; 26844e704ee3SHerbert Xu if (unlikely(skb->len <= mss)) 26854e704ee3SHerbert Xu goto out; 26864e704ee3SHerbert Xu 26873820c3f3SHerbert Xu if (skb_gso_ok(skb, features | NETIF_F_GSO_ROBUST)) { 26883820c3f3SHerbert Xu /* Packet is from an untrusted source, reset gso_segs. */ 2689bbcf467dSHerbert Xu int type = skb_shinfo(skb)->gso_type; 26903820c3f3SHerbert Xu 2691bbcf467dSHerbert Xu if (unlikely(type & 2692bbcf467dSHerbert Xu ~(SKB_GSO_TCPV4 | 2693bbcf467dSHerbert Xu SKB_GSO_DODGY | 2694bbcf467dSHerbert Xu SKB_GSO_TCP_ECN | 2695bbcf467dSHerbert Xu SKB_GSO_TCPV6 | 2696bbcf467dSHerbert Xu 0) || 2697bbcf467dSHerbert Xu !(type & (SKB_GSO_TCPV4 | SKB_GSO_TCPV6)))) 2698bbcf467dSHerbert Xu goto out; 2699bbcf467dSHerbert Xu 2700172589ccSIlpo Järvinen skb_shinfo(skb)->gso_segs = DIV_ROUND_UP(skb->len, mss); 27013820c3f3SHerbert Xu 27023820c3f3SHerbert Xu segs = NULL; 27033820c3f3SHerbert Xu goto out; 27043820c3f3SHerbert Xu } 27053820c3f3SHerbert Xu 2706576a30ebSHerbert Xu segs = skb_segment(skb, features); 2707f4c50d99SHerbert Xu if (IS_ERR(segs)) 2708f4c50d99SHerbert Xu goto out; 2709f4c50d99SHerbert Xu 27104e704ee3SHerbert Xu delta = htonl(oldlen + (thlen + mss)); 2711f4c50d99SHerbert Xu 2712f4c50d99SHerbert Xu skb = segs; 2713aa8223c7SArnaldo Carvalho de Melo th = tcp_hdr(skb); 2714f4c50d99SHerbert Xu seq = ntohl(th->seq); 2715f4c50d99SHerbert Xu 2716f4c50d99SHerbert Xu do { 2717f4c50d99SHerbert Xu th->fin = th->psh = 0; 2718f4c50d99SHerbert Xu 2719d3bc23e7SAl Viro th->check = ~csum_fold((__force __wsum)((__force u32)th->check + 2720d3bc23e7SAl Viro (__force u32)delta)); 272184fa7933SPatrick McHardy if (skb->ip_summed != CHECKSUM_PARTIAL) 27229c70220bSArnaldo Carvalho de Melo th->check = 27239c70220bSArnaldo Carvalho de Melo csum_fold(csum_partial(skb_transport_header(skb), 27249c70220bSArnaldo Carvalho de Melo thlen, skb->csum)); 2725f4c50d99SHerbert Xu 27264e704ee3SHerbert Xu seq += mss; 2727f4c50d99SHerbert Xu skb = skb->next; 2728aa8223c7SArnaldo Carvalho de Melo th = tcp_hdr(skb); 2729f4c50d99SHerbert Xu 2730f4c50d99SHerbert Xu th->seq = htonl(seq); 2731f4c50d99SHerbert Xu th->cwr = 0; 2732f4c50d99SHerbert Xu } while (skb->next); 2733f4c50d99SHerbert Xu 273427a884dcSArnaldo Carvalho de Melo delta = htonl(oldlen + (skb->tail - skb->transport_header) + 27359c70220bSArnaldo Carvalho de Melo skb->data_len); 2736d3bc23e7SAl Viro th->check = ~csum_fold((__force __wsum)((__force u32)th->check + 2737d3bc23e7SAl Viro (__force u32)delta)); 273884fa7933SPatrick McHardy if (skb->ip_summed != CHECKSUM_PARTIAL) 27399c70220bSArnaldo Carvalho de Melo th->check = csum_fold(csum_partial(skb_transport_header(skb), 27409c70220bSArnaldo Carvalho de Melo thlen, skb->csum)); 2741f4c50d99SHerbert Xu 2742f4c50d99SHerbert Xu out: 2743f4c50d99SHerbert Xu return segs; 2744f4c50d99SHerbert Xu } 2745adcfc7d0SHerbert Xu EXPORT_SYMBOL(tcp_tso_segment); 2746f4c50d99SHerbert Xu 2747bf296b12SHerbert Xu struct sk_buff **tcp_gro_receive(struct sk_buff **head, struct sk_buff *skb) 2748bf296b12SHerbert Xu { 2749bf296b12SHerbert Xu struct sk_buff **pp = NULL; 2750bf296b12SHerbert Xu struct sk_buff *p; 2751bf296b12SHerbert Xu struct tcphdr *th; 2752bf296b12SHerbert Xu struct tcphdr *th2; 2753a0a69a01SHerbert Xu unsigned int len; 2754bf296b12SHerbert Xu unsigned int thlen; 27550eae88f3SEric Dumazet __be32 flags; 2756bf296b12SHerbert Xu unsigned int mss = 1; 2757a5b1cf28SHerbert Xu unsigned int hlen; 2758a5b1cf28SHerbert Xu unsigned int off; 2759bf296b12SHerbert Xu int flush = 1; 2760aa6320d3SHerbert Xu int i; 2761bf296b12SHerbert Xu 2762a5b1cf28SHerbert Xu off = skb_gro_offset(skb); 2763a5b1cf28SHerbert Xu hlen = off + sizeof(*th); 2764a5b1cf28SHerbert Xu th = skb_gro_header_fast(skb, off); 2765a5b1cf28SHerbert Xu if (skb_gro_header_hard(skb, hlen)) { 2766a5b1cf28SHerbert Xu th = skb_gro_header_slow(skb, hlen, off); 276786911732SHerbert Xu if (unlikely(!th)) 2768bf296b12SHerbert Xu goto out; 2769a5b1cf28SHerbert Xu } 2770bf296b12SHerbert Xu 2771bf296b12SHerbert Xu thlen = th->doff * 4; 2772bf296b12SHerbert Xu if (thlen < sizeof(*th)) 2773bf296b12SHerbert Xu goto out; 2774bf296b12SHerbert Xu 2775a5b1cf28SHerbert Xu hlen = off + thlen; 2776a5b1cf28SHerbert Xu if (skb_gro_header_hard(skb, hlen)) { 2777a5b1cf28SHerbert Xu th = skb_gro_header_slow(skb, hlen, off); 277886911732SHerbert Xu if (unlikely(!th)) 2779bf296b12SHerbert Xu goto out; 2780a5b1cf28SHerbert Xu } 2781bf296b12SHerbert Xu 278286911732SHerbert Xu skb_gro_pull(skb, thlen); 2783bf296b12SHerbert Xu 2784a0a69a01SHerbert Xu len = skb_gro_len(skb); 2785bf296b12SHerbert Xu flags = tcp_flag_word(th); 2786bf296b12SHerbert Xu 2787bf296b12SHerbert Xu for (; (p = *head); head = &p->next) { 2788bf296b12SHerbert Xu if (!NAPI_GRO_CB(p)->same_flow) 2789bf296b12SHerbert Xu continue; 2790bf296b12SHerbert Xu 2791bf296b12SHerbert Xu th2 = tcp_hdr(p); 2792bf296b12SHerbert Xu 2793745898eaSHerbert Xu if (*(u32 *)&th->source ^ *(u32 *)&th2->source) { 2794bf296b12SHerbert Xu NAPI_GRO_CB(p)->same_flow = 0; 2795bf296b12SHerbert Xu continue; 2796bf296b12SHerbert Xu } 2797bf296b12SHerbert Xu 2798bf296b12SHerbert Xu goto found; 2799bf296b12SHerbert Xu } 2800bf296b12SHerbert Xu 2801bf296b12SHerbert Xu goto out_check_final; 2802bf296b12SHerbert Xu 2803bf296b12SHerbert Xu found: 2804bf296b12SHerbert Xu flush = NAPI_GRO_CB(p)->flush; 28050eae88f3SEric Dumazet flush |= (__force int)(flags & TCP_FLAG_CWR); 28060eae88f3SEric Dumazet flush |= (__force int)((flags ^ tcp_flag_word(th2)) & 28070eae88f3SEric Dumazet ~(TCP_FLAG_CWR | TCP_FLAG_FIN | TCP_FLAG_PSH)); 28080eae88f3SEric Dumazet flush |= (__force int)(th->ack_seq ^ th2->ack_seq); 2809a2a804cdSHerbert Xu for (i = sizeof(*th); i < thlen; i += 4) 2810aa6320d3SHerbert Xu flush |= *(u32 *)((u8 *)th + i) ^ 2811aa6320d3SHerbert Xu *(u32 *)((u8 *)th2 + i); 2812bf296b12SHerbert Xu 2813b530256dSHerbert Xu mss = skb_shinfo(p)->gso_size; 2814bf296b12SHerbert Xu 281530a3ae30SHerbert Xu flush |= (len - 1) >= mss; 2816aa6320d3SHerbert Xu flush |= (ntohl(th2->seq) + skb_gro_len(p)) ^ ntohl(th->seq); 2817bf296b12SHerbert Xu 2818bf296b12SHerbert Xu if (flush || skb_gro_receive(head, skb)) { 2819bf296b12SHerbert Xu mss = 1; 2820bf296b12SHerbert Xu goto out_check_final; 2821bf296b12SHerbert Xu } 2822bf296b12SHerbert Xu 2823bf296b12SHerbert Xu p = *head; 2824bf296b12SHerbert Xu th2 = tcp_hdr(p); 2825bf296b12SHerbert Xu tcp_flag_word(th2) |= flags & (TCP_FLAG_FIN | TCP_FLAG_PSH); 2826bf296b12SHerbert Xu 2827bf296b12SHerbert Xu out_check_final: 2828a0a69a01SHerbert Xu flush = len < mss; 28290eae88f3SEric Dumazet flush |= (__force int)(flags & (TCP_FLAG_URG | TCP_FLAG_PSH | 28300eae88f3SEric Dumazet TCP_FLAG_RST | TCP_FLAG_SYN | 28310eae88f3SEric Dumazet TCP_FLAG_FIN)); 2832bf296b12SHerbert Xu 2833bf296b12SHerbert Xu if (p && (!NAPI_GRO_CB(skb)->same_flow || flush)) 2834bf296b12SHerbert Xu pp = head; 2835bf296b12SHerbert Xu 2836bf296b12SHerbert Xu out: 2837bf296b12SHerbert Xu NAPI_GRO_CB(skb)->flush |= flush; 2838bf296b12SHerbert Xu 2839bf296b12SHerbert Xu return pp; 2840bf296b12SHerbert Xu } 2841684f2176SHerbert Xu EXPORT_SYMBOL(tcp_gro_receive); 2842bf296b12SHerbert Xu 2843bf296b12SHerbert Xu int tcp_gro_complete(struct sk_buff *skb) 2844bf296b12SHerbert Xu { 2845bf296b12SHerbert Xu struct tcphdr *th = tcp_hdr(skb); 2846bf296b12SHerbert Xu 2847bf296b12SHerbert Xu skb->csum_start = skb_transport_header(skb) - skb->head; 2848bf296b12SHerbert Xu skb->csum_offset = offsetof(struct tcphdr, check); 2849bf296b12SHerbert Xu skb->ip_summed = CHECKSUM_PARTIAL; 2850bf296b12SHerbert Xu 2851bf296b12SHerbert Xu skb_shinfo(skb)->gso_segs = NAPI_GRO_CB(skb)->count; 2852bf296b12SHerbert Xu 2853bf296b12SHerbert Xu if (th->cwr) 2854bf296b12SHerbert Xu skb_shinfo(skb)->gso_type |= SKB_GSO_TCP_ECN; 2855bf296b12SHerbert Xu 2856bf296b12SHerbert Xu return 0; 2857bf296b12SHerbert Xu } 2858684f2176SHerbert Xu EXPORT_SYMBOL(tcp_gro_complete); 2859bf296b12SHerbert Xu 2860cfb6eeb4SYOSHIFUJI Hideaki #ifdef CONFIG_TCP_MD5SIG 2861cfb6eeb4SYOSHIFUJI Hideaki static unsigned long tcp_md5sig_users; 2862765cf997SEric Dumazet static struct tcp_md5sig_pool __percpu *tcp_md5sig_pool; 2863cfb6eeb4SYOSHIFUJI Hideaki static DEFINE_SPINLOCK(tcp_md5sig_pool_lock); 2864cfb6eeb4SYOSHIFUJI Hideaki 2865765cf997SEric Dumazet static void __tcp_free_md5sig_pool(struct tcp_md5sig_pool __percpu *pool) 2866cfb6eeb4SYOSHIFUJI Hideaki { 2867cfb6eeb4SYOSHIFUJI Hideaki int cpu; 2868765cf997SEric Dumazet 2869cfb6eeb4SYOSHIFUJI Hideaki for_each_possible_cpu(cpu) { 2870765cf997SEric Dumazet struct tcp_md5sig_pool *p = per_cpu_ptr(pool, cpu); 2871765cf997SEric Dumazet 2872cfb6eeb4SYOSHIFUJI Hideaki if (p->md5_desc.tfm) 2873cfb6eeb4SYOSHIFUJI Hideaki crypto_free_hash(p->md5_desc.tfm); 2874cfb6eeb4SYOSHIFUJI Hideaki } 2875cfb6eeb4SYOSHIFUJI Hideaki free_percpu(pool); 2876cfb6eeb4SYOSHIFUJI Hideaki } 2877cfb6eeb4SYOSHIFUJI Hideaki 2878cfb6eeb4SYOSHIFUJI Hideaki void tcp_free_md5sig_pool(void) 2879cfb6eeb4SYOSHIFUJI Hideaki { 2880765cf997SEric Dumazet struct tcp_md5sig_pool __percpu *pool = NULL; 2881cfb6eeb4SYOSHIFUJI Hideaki 28822c4f6219SDavid S. Miller spin_lock_bh(&tcp_md5sig_pool_lock); 2883cfb6eeb4SYOSHIFUJI Hideaki if (--tcp_md5sig_users == 0) { 2884cfb6eeb4SYOSHIFUJI Hideaki pool = tcp_md5sig_pool; 2885cfb6eeb4SYOSHIFUJI Hideaki tcp_md5sig_pool = NULL; 2886cfb6eeb4SYOSHIFUJI Hideaki } 28872c4f6219SDavid S. Miller spin_unlock_bh(&tcp_md5sig_pool_lock); 2888cfb6eeb4SYOSHIFUJI Hideaki if (pool) 2889cfb6eeb4SYOSHIFUJI Hideaki __tcp_free_md5sig_pool(pool); 2890cfb6eeb4SYOSHIFUJI Hideaki } 2891cfb6eeb4SYOSHIFUJI Hideaki EXPORT_SYMBOL(tcp_free_md5sig_pool); 2892cfb6eeb4SYOSHIFUJI Hideaki 2893765cf997SEric Dumazet static struct tcp_md5sig_pool __percpu * 28947d720c3eSTejun Heo __tcp_alloc_md5sig_pool(struct sock *sk) 2895cfb6eeb4SYOSHIFUJI Hideaki { 2896cfb6eeb4SYOSHIFUJI Hideaki int cpu; 2897765cf997SEric Dumazet struct tcp_md5sig_pool __percpu *pool; 2898cfb6eeb4SYOSHIFUJI Hideaki 2899765cf997SEric Dumazet pool = alloc_percpu(struct tcp_md5sig_pool); 2900cfb6eeb4SYOSHIFUJI Hideaki if (!pool) 2901cfb6eeb4SYOSHIFUJI Hideaki return NULL; 2902cfb6eeb4SYOSHIFUJI Hideaki 2903cfb6eeb4SYOSHIFUJI Hideaki for_each_possible_cpu(cpu) { 2904cfb6eeb4SYOSHIFUJI Hideaki struct crypto_hash *hash; 2905cfb6eeb4SYOSHIFUJI Hideaki 2906cfb6eeb4SYOSHIFUJI Hideaki hash = crypto_alloc_hash("md5", 0, CRYPTO_ALG_ASYNC); 2907cfb6eeb4SYOSHIFUJI Hideaki if (!hash || IS_ERR(hash)) 2908cfb6eeb4SYOSHIFUJI Hideaki goto out_free; 2909cfb6eeb4SYOSHIFUJI Hideaki 2910765cf997SEric Dumazet per_cpu_ptr(pool, cpu)->md5_desc.tfm = hash; 2911cfb6eeb4SYOSHIFUJI Hideaki } 2912cfb6eeb4SYOSHIFUJI Hideaki return pool; 2913cfb6eeb4SYOSHIFUJI Hideaki out_free: 2914cfb6eeb4SYOSHIFUJI Hideaki __tcp_free_md5sig_pool(pool); 2915cfb6eeb4SYOSHIFUJI Hideaki return NULL; 2916cfb6eeb4SYOSHIFUJI Hideaki } 2917cfb6eeb4SYOSHIFUJI Hideaki 2918765cf997SEric Dumazet struct tcp_md5sig_pool __percpu *tcp_alloc_md5sig_pool(struct sock *sk) 2919cfb6eeb4SYOSHIFUJI Hideaki { 2920765cf997SEric Dumazet struct tcp_md5sig_pool __percpu *pool; 2921cfb6eeb4SYOSHIFUJI Hideaki int alloc = 0; 2922cfb6eeb4SYOSHIFUJI Hideaki 2923cfb6eeb4SYOSHIFUJI Hideaki retry: 29242c4f6219SDavid S. Miller spin_lock_bh(&tcp_md5sig_pool_lock); 2925cfb6eeb4SYOSHIFUJI Hideaki pool = tcp_md5sig_pool; 2926cfb6eeb4SYOSHIFUJI Hideaki if (tcp_md5sig_users++ == 0) { 2927cfb6eeb4SYOSHIFUJI Hideaki alloc = 1; 29282c4f6219SDavid S. Miller spin_unlock_bh(&tcp_md5sig_pool_lock); 2929cfb6eeb4SYOSHIFUJI Hideaki } else if (!pool) { 2930cfb6eeb4SYOSHIFUJI Hideaki tcp_md5sig_users--; 29312c4f6219SDavid S. Miller spin_unlock_bh(&tcp_md5sig_pool_lock); 2932cfb6eeb4SYOSHIFUJI Hideaki cpu_relax(); 2933cfb6eeb4SYOSHIFUJI Hideaki goto retry; 2934cfb6eeb4SYOSHIFUJI Hideaki } else 29352c4f6219SDavid S. Miller spin_unlock_bh(&tcp_md5sig_pool_lock); 2936cfb6eeb4SYOSHIFUJI Hideaki 2937cfb6eeb4SYOSHIFUJI Hideaki if (alloc) { 2938cfb6eeb4SYOSHIFUJI Hideaki /* we cannot hold spinlock here because this may sleep. */ 2939765cf997SEric Dumazet struct tcp_md5sig_pool __percpu *p; 29407d720c3eSTejun Heo 29417d720c3eSTejun Heo p = __tcp_alloc_md5sig_pool(sk); 29422c4f6219SDavid S. Miller spin_lock_bh(&tcp_md5sig_pool_lock); 2943cfb6eeb4SYOSHIFUJI Hideaki if (!p) { 2944cfb6eeb4SYOSHIFUJI Hideaki tcp_md5sig_users--; 29452c4f6219SDavid S. Miller spin_unlock_bh(&tcp_md5sig_pool_lock); 2946cfb6eeb4SYOSHIFUJI Hideaki return NULL; 2947cfb6eeb4SYOSHIFUJI Hideaki } 2948cfb6eeb4SYOSHIFUJI Hideaki pool = tcp_md5sig_pool; 2949cfb6eeb4SYOSHIFUJI Hideaki if (pool) { 2950cfb6eeb4SYOSHIFUJI Hideaki /* oops, it has already been assigned. */ 29512c4f6219SDavid S. Miller spin_unlock_bh(&tcp_md5sig_pool_lock); 2952cfb6eeb4SYOSHIFUJI Hideaki __tcp_free_md5sig_pool(p); 2953cfb6eeb4SYOSHIFUJI Hideaki } else { 2954cfb6eeb4SYOSHIFUJI Hideaki tcp_md5sig_pool = pool = p; 29552c4f6219SDavid S. Miller spin_unlock_bh(&tcp_md5sig_pool_lock); 2956cfb6eeb4SYOSHIFUJI Hideaki } 2957cfb6eeb4SYOSHIFUJI Hideaki } 2958cfb6eeb4SYOSHIFUJI Hideaki return pool; 2959cfb6eeb4SYOSHIFUJI Hideaki } 2960cfb6eeb4SYOSHIFUJI Hideaki EXPORT_SYMBOL(tcp_alloc_md5sig_pool); 2961cfb6eeb4SYOSHIFUJI Hideaki 296235790c04SEric Dumazet 296335790c04SEric Dumazet /** 296435790c04SEric Dumazet * tcp_get_md5sig_pool - get md5sig_pool for this user 296535790c04SEric Dumazet * 296635790c04SEric Dumazet * We use percpu structure, so if we succeed, we exit with preemption 296735790c04SEric Dumazet * and BH disabled, to make sure another thread or softirq handling 296835790c04SEric Dumazet * wont try to get same context. 296935790c04SEric Dumazet */ 297035790c04SEric Dumazet struct tcp_md5sig_pool *tcp_get_md5sig_pool(void) 2971cfb6eeb4SYOSHIFUJI Hideaki { 2972765cf997SEric Dumazet struct tcp_md5sig_pool __percpu *p; 297335790c04SEric Dumazet 297435790c04SEric Dumazet local_bh_disable(); 297535790c04SEric Dumazet 297635790c04SEric Dumazet spin_lock(&tcp_md5sig_pool_lock); 2977cfb6eeb4SYOSHIFUJI Hideaki p = tcp_md5sig_pool; 2978cfb6eeb4SYOSHIFUJI Hideaki if (p) 2979cfb6eeb4SYOSHIFUJI Hideaki tcp_md5sig_users++; 298035790c04SEric Dumazet spin_unlock(&tcp_md5sig_pool_lock); 298135790c04SEric Dumazet 298235790c04SEric Dumazet if (p) 2983765cf997SEric Dumazet return this_cpu_ptr(p); 298435790c04SEric Dumazet 298535790c04SEric Dumazet local_bh_enable(); 298635790c04SEric Dumazet return NULL; 2987cfb6eeb4SYOSHIFUJI Hideaki } 298835790c04SEric Dumazet EXPORT_SYMBOL(tcp_get_md5sig_pool); 2989cfb6eeb4SYOSHIFUJI Hideaki 299035790c04SEric Dumazet void tcp_put_md5sig_pool(void) 29916931ba7cSDavid S. Miller { 299235790c04SEric Dumazet local_bh_enable(); 29936931ba7cSDavid S. Miller tcp_free_md5sig_pool(); 2994cfb6eeb4SYOSHIFUJI Hideaki } 299535790c04SEric Dumazet EXPORT_SYMBOL(tcp_put_md5sig_pool); 299649a72dfbSAdam Langley 299749a72dfbSAdam Langley int tcp_md5_hash_header(struct tcp_md5sig_pool *hp, 299849a72dfbSAdam Langley struct tcphdr *th) 299949a72dfbSAdam Langley { 300049a72dfbSAdam Langley struct scatterlist sg; 300149a72dfbSAdam Langley int err; 300249a72dfbSAdam Langley 300349a72dfbSAdam Langley __sum16 old_checksum = th->check; 300449a72dfbSAdam Langley th->check = 0; 300549a72dfbSAdam Langley /* options aren't included in the hash */ 300649a72dfbSAdam Langley sg_init_one(&sg, th, sizeof(struct tcphdr)); 300749a72dfbSAdam Langley err = crypto_hash_update(&hp->md5_desc, &sg, sizeof(struct tcphdr)); 300849a72dfbSAdam Langley th->check = old_checksum; 300949a72dfbSAdam Langley return err; 301049a72dfbSAdam Langley } 301149a72dfbSAdam Langley EXPORT_SYMBOL(tcp_md5_hash_header); 301249a72dfbSAdam Langley 301349a72dfbSAdam Langley int tcp_md5_hash_skb_data(struct tcp_md5sig_pool *hp, 301449a72dfbSAdam Langley struct sk_buff *skb, unsigned header_len) 301549a72dfbSAdam Langley { 301649a72dfbSAdam Langley struct scatterlist sg; 301749a72dfbSAdam Langley const struct tcphdr *tp = tcp_hdr(skb); 301849a72dfbSAdam Langley struct hash_desc *desc = &hp->md5_desc; 301949a72dfbSAdam Langley unsigned i; 302049a72dfbSAdam Langley const unsigned head_data_len = skb_headlen(skb) > header_len ? 302149a72dfbSAdam Langley skb_headlen(skb) - header_len : 0; 302249a72dfbSAdam Langley const struct skb_shared_info *shi = skb_shinfo(skb); 3023d7fd1b57SEric Dumazet struct sk_buff *frag_iter; 302449a72dfbSAdam Langley 302549a72dfbSAdam Langley sg_init_table(&sg, 1); 302649a72dfbSAdam Langley 302749a72dfbSAdam Langley sg_set_buf(&sg, ((u8 *) tp) + header_len, head_data_len); 302849a72dfbSAdam Langley if (crypto_hash_update(desc, &sg, head_data_len)) 302949a72dfbSAdam Langley return 1; 303049a72dfbSAdam Langley 303149a72dfbSAdam Langley for (i = 0; i < shi->nr_frags; ++i) { 303249a72dfbSAdam Langley const struct skb_frag_struct *f = &shi->frags[i]; 3033aff65da0SIan Campbell struct page *page = skb_frag_page(f); 3034aff65da0SIan Campbell sg_set_page(&sg, page, f->size, f->page_offset); 303549a72dfbSAdam Langley if (crypto_hash_update(desc, &sg, f->size)) 303649a72dfbSAdam Langley return 1; 303749a72dfbSAdam Langley } 303849a72dfbSAdam Langley 3039d7fd1b57SEric Dumazet skb_walk_frags(skb, frag_iter) 3040d7fd1b57SEric Dumazet if (tcp_md5_hash_skb_data(hp, frag_iter, 0)) 3041d7fd1b57SEric Dumazet return 1; 3042d7fd1b57SEric Dumazet 304349a72dfbSAdam Langley return 0; 304449a72dfbSAdam Langley } 304549a72dfbSAdam Langley EXPORT_SYMBOL(tcp_md5_hash_skb_data); 304649a72dfbSAdam Langley 304749a72dfbSAdam Langley int tcp_md5_hash_key(struct tcp_md5sig_pool *hp, struct tcp_md5sig_key *key) 304849a72dfbSAdam Langley { 304949a72dfbSAdam Langley struct scatterlist sg; 305049a72dfbSAdam Langley 305149a72dfbSAdam Langley sg_init_one(&sg, key->key, key->keylen); 305249a72dfbSAdam Langley return crypto_hash_update(&hp->md5_desc, &sg, key->keylen); 305349a72dfbSAdam Langley } 305449a72dfbSAdam Langley EXPORT_SYMBOL(tcp_md5_hash_key); 305549a72dfbSAdam Langley 3056cfb6eeb4SYOSHIFUJI Hideaki #endif 3057cfb6eeb4SYOSHIFUJI Hideaki 3058da5c78c8SWilliam Allen Simpson /** 3059da5c78c8SWilliam Allen Simpson * Each Responder maintains up to two secret values concurrently for 3060da5c78c8SWilliam Allen Simpson * efficient secret rollover. Each secret value has 4 states: 3061da5c78c8SWilliam Allen Simpson * 3062da5c78c8SWilliam Allen Simpson * Generating. (tcp_secret_generating != tcp_secret_primary) 3063da5c78c8SWilliam Allen Simpson * Generates new Responder-Cookies, but not yet used for primary 3064da5c78c8SWilliam Allen Simpson * verification. This is a short-term state, typically lasting only 3065da5c78c8SWilliam Allen Simpson * one round trip time (RTT). 3066da5c78c8SWilliam Allen Simpson * 3067da5c78c8SWilliam Allen Simpson * Primary. (tcp_secret_generating == tcp_secret_primary) 3068da5c78c8SWilliam Allen Simpson * Used both for generation and primary verification. 3069da5c78c8SWilliam Allen Simpson * 3070da5c78c8SWilliam Allen Simpson * Retiring. (tcp_secret_retiring != tcp_secret_secondary) 3071da5c78c8SWilliam Allen Simpson * Used for verification, until the first failure that can be 3072da5c78c8SWilliam Allen Simpson * verified by the newer Generating secret. At that time, this 3073da5c78c8SWilliam Allen Simpson * cookie's state is changed to Secondary, and the Generating 3074da5c78c8SWilliam Allen Simpson * cookie's state is changed to Primary. This is a short-term state, 3075da5c78c8SWilliam Allen Simpson * typically lasting only one round trip time (RTT). 3076da5c78c8SWilliam Allen Simpson * 3077da5c78c8SWilliam Allen Simpson * Secondary. (tcp_secret_retiring == tcp_secret_secondary) 3078da5c78c8SWilliam Allen Simpson * Used for secondary verification, after primary verification 3079da5c78c8SWilliam Allen Simpson * failures. This state lasts no more than twice the Maximum Segment 3080da5c78c8SWilliam Allen Simpson * Lifetime (2MSL). Then, the secret is discarded. 3081da5c78c8SWilliam Allen Simpson */ 3082da5c78c8SWilliam Allen Simpson struct tcp_cookie_secret { 3083da5c78c8SWilliam Allen Simpson /* The secret is divided into two parts. The digest part is the 3084da5c78c8SWilliam Allen Simpson * equivalent of previously hashing a secret and saving the state, 3085da5c78c8SWilliam Allen Simpson * and serves as an initialization vector (IV). The message part 3086da5c78c8SWilliam Allen Simpson * serves as the trailing secret. 3087da5c78c8SWilliam Allen Simpson */ 3088da5c78c8SWilliam Allen Simpson u32 secrets[COOKIE_WORKSPACE_WORDS]; 3089da5c78c8SWilliam Allen Simpson unsigned long expires; 3090da5c78c8SWilliam Allen Simpson }; 3091da5c78c8SWilliam Allen Simpson 3092da5c78c8SWilliam Allen Simpson #define TCP_SECRET_1MSL (HZ * TCP_PAWS_MSL) 3093da5c78c8SWilliam Allen Simpson #define TCP_SECRET_2MSL (HZ * TCP_PAWS_MSL * 2) 3094da5c78c8SWilliam Allen Simpson #define TCP_SECRET_LIFE (HZ * 600) 3095da5c78c8SWilliam Allen Simpson 3096da5c78c8SWilliam Allen Simpson static struct tcp_cookie_secret tcp_secret_one; 3097da5c78c8SWilliam Allen Simpson static struct tcp_cookie_secret tcp_secret_two; 3098da5c78c8SWilliam Allen Simpson 3099da5c78c8SWilliam Allen Simpson /* Essentially a circular list, without dynamic allocation. */ 3100da5c78c8SWilliam Allen Simpson static struct tcp_cookie_secret *tcp_secret_generating; 3101da5c78c8SWilliam Allen Simpson static struct tcp_cookie_secret *tcp_secret_primary; 3102da5c78c8SWilliam Allen Simpson static struct tcp_cookie_secret *tcp_secret_retiring; 3103da5c78c8SWilliam Allen Simpson static struct tcp_cookie_secret *tcp_secret_secondary; 3104da5c78c8SWilliam Allen Simpson 3105da5c78c8SWilliam Allen Simpson static DEFINE_SPINLOCK(tcp_secret_locker); 3106da5c78c8SWilliam Allen Simpson 3107da5c78c8SWilliam Allen Simpson /* Select a pseudo-random word in the cookie workspace. 3108da5c78c8SWilliam Allen Simpson */ 3109da5c78c8SWilliam Allen Simpson static inline u32 tcp_cookie_work(const u32 *ws, const int n) 3110da5c78c8SWilliam Allen Simpson { 3111da5c78c8SWilliam Allen Simpson return ws[COOKIE_DIGEST_WORDS + ((COOKIE_MESSAGE_WORDS-1) & ws[n])]; 3112da5c78c8SWilliam Allen Simpson } 3113da5c78c8SWilliam Allen Simpson 3114da5c78c8SWilliam Allen Simpson /* Fill bakery[COOKIE_WORKSPACE_WORDS] with generator, updating as needed. 3115da5c78c8SWilliam Allen Simpson * Called in softirq context. 3116da5c78c8SWilliam Allen Simpson * Returns: 0 for success. 3117da5c78c8SWilliam Allen Simpson */ 3118da5c78c8SWilliam Allen Simpson int tcp_cookie_generator(u32 *bakery) 3119da5c78c8SWilliam Allen Simpson { 3120da5c78c8SWilliam Allen Simpson unsigned long jiffy = jiffies; 3121da5c78c8SWilliam Allen Simpson 3122da5c78c8SWilliam Allen Simpson if (unlikely(time_after_eq(jiffy, tcp_secret_generating->expires))) { 3123da5c78c8SWilliam Allen Simpson spin_lock_bh(&tcp_secret_locker); 3124da5c78c8SWilliam Allen Simpson if (!time_after_eq(jiffy, tcp_secret_generating->expires)) { 3125da5c78c8SWilliam Allen Simpson /* refreshed by another */ 3126da5c78c8SWilliam Allen Simpson memcpy(bakery, 3127da5c78c8SWilliam Allen Simpson &tcp_secret_generating->secrets[0], 3128da5c78c8SWilliam Allen Simpson COOKIE_WORKSPACE_WORDS); 3129da5c78c8SWilliam Allen Simpson } else { 3130da5c78c8SWilliam Allen Simpson /* still needs refreshing */ 3131da5c78c8SWilliam Allen Simpson get_random_bytes(bakery, COOKIE_WORKSPACE_WORDS); 3132da5c78c8SWilliam Allen Simpson 3133da5c78c8SWilliam Allen Simpson /* The first time, paranoia assumes that the 3134da5c78c8SWilliam Allen Simpson * randomization function isn't as strong. But, 3135da5c78c8SWilliam Allen Simpson * this secret initialization is delayed until 3136da5c78c8SWilliam Allen Simpson * the last possible moment (packet arrival). 3137da5c78c8SWilliam Allen Simpson * Although that time is observable, it is 3138da5c78c8SWilliam Allen Simpson * unpredictably variable. Mash in the most 3139da5c78c8SWilliam Allen Simpson * volatile clock bits available, and expire the 3140da5c78c8SWilliam Allen Simpson * secret extra quickly. 3141da5c78c8SWilliam Allen Simpson */ 3142da5c78c8SWilliam Allen Simpson if (unlikely(tcp_secret_primary->expires == 3143da5c78c8SWilliam Allen Simpson tcp_secret_secondary->expires)) { 3144da5c78c8SWilliam Allen Simpson struct timespec tv; 3145da5c78c8SWilliam Allen Simpson 3146da5c78c8SWilliam Allen Simpson getnstimeofday(&tv); 3147da5c78c8SWilliam Allen Simpson bakery[COOKIE_DIGEST_WORDS+0] ^= 3148da5c78c8SWilliam Allen Simpson (u32)tv.tv_nsec; 3149da5c78c8SWilliam Allen Simpson 3150da5c78c8SWilliam Allen Simpson tcp_secret_secondary->expires = jiffy 3151da5c78c8SWilliam Allen Simpson + TCP_SECRET_1MSL 3152da5c78c8SWilliam Allen Simpson + (0x0f & tcp_cookie_work(bakery, 0)); 3153da5c78c8SWilliam Allen Simpson } else { 3154da5c78c8SWilliam Allen Simpson tcp_secret_secondary->expires = jiffy 3155da5c78c8SWilliam Allen Simpson + TCP_SECRET_LIFE 3156da5c78c8SWilliam Allen Simpson + (0xff & tcp_cookie_work(bakery, 1)); 3157da5c78c8SWilliam Allen Simpson tcp_secret_primary->expires = jiffy 3158da5c78c8SWilliam Allen Simpson + TCP_SECRET_2MSL 3159da5c78c8SWilliam Allen Simpson + (0x1f & tcp_cookie_work(bakery, 2)); 3160da5c78c8SWilliam Allen Simpson } 3161da5c78c8SWilliam Allen Simpson memcpy(&tcp_secret_secondary->secrets[0], 3162da5c78c8SWilliam Allen Simpson bakery, COOKIE_WORKSPACE_WORDS); 3163da5c78c8SWilliam Allen Simpson 3164da5c78c8SWilliam Allen Simpson rcu_assign_pointer(tcp_secret_generating, 3165da5c78c8SWilliam Allen Simpson tcp_secret_secondary); 3166da5c78c8SWilliam Allen Simpson rcu_assign_pointer(tcp_secret_retiring, 3167da5c78c8SWilliam Allen Simpson tcp_secret_primary); 3168da5c78c8SWilliam Allen Simpson /* 3169da5c78c8SWilliam Allen Simpson * Neither call_rcu() nor synchronize_rcu() needed. 3170da5c78c8SWilliam Allen Simpson * Retiring data is not freed. It is replaced after 3171da5c78c8SWilliam Allen Simpson * further (locked) pointer updates, and a quiet time 3172da5c78c8SWilliam Allen Simpson * (minimum 1MSL, maximum LIFE - 2MSL). 3173da5c78c8SWilliam Allen Simpson */ 3174da5c78c8SWilliam Allen Simpson } 3175da5c78c8SWilliam Allen Simpson spin_unlock_bh(&tcp_secret_locker); 3176da5c78c8SWilliam Allen Simpson } else { 3177da5c78c8SWilliam Allen Simpson rcu_read_lock_bh(); 3178da5c78c8SWilliam Allen Simpson memcpy(bakery, 3179da5c78c8SWilliam Allen Simpson &rcu_dereference(tcp_secret_generating)->secrets[0], 3180da5c78c8SWilliam Allen Simpson COOKIE_WORKSPACE_WORDS); 3181da5c78c8SWilliam Allen Simpson rcu_read_unlock_bh(); 3182da5c78c8SWilliam Allen Simpson } 3183da5c78c8SWilliam Allen Simpson return 0; 3184da5c78c8SWilliam Allen Simpson } 3185da5c78c8SWilliam Allen Simpson EXPORT_SYMBOL(tcp_cookie_generator); 3186da5c78c8SWilliam Allen Simpson 31874ac02babSAndi Kleen void tcp_done(struct sock *sk) 31884ac02babSAndi Kleen { 31894ac02babSAndi Kleen if (sk->sk_state == TCP_SYN_SENT || sk->sk_state == TCP_SYN_RECV) 319063231bddSPavel Emelyanov TCP_INC_STATS_BH(sock_net(sk), TCP_MIB_ATTEMPTFAILS); 31914ac02babSAndi Kleen 31924ac02babSAndi Kleen tcp_set_state(sk, TCP_CLOSE); 31934ac02babSAndi Kleen tcp_clear_xmit_timers(sk); 31944ac02babSAndi Kleen 31954ac02babSAndi Kleen sk->sk_shutdown = SHUTDOWN_MASK; 31964ac02babSAndi Kleen 31974ac02babSAndi Kleen if (!sock_flag(sk, SOCK_DEAD)) 31984ac02babSAndi Kleen sk->sk_state_change(sk); 31994ac02babSAndi Kleen else 32004ac02babSAndi Kleen inet_csk_destroy_sock(sk); 32014ac02babSAndi Kleen } 32024ac02babSAndi Kleen EXPORT_SYMBOL_GPL(tcp_done); 32034ac02babSAndi Kleen 32045f8ef48dSStephen Hemminger extern struct tcp_congestion_ops tcp_reno; 32051da177e4SLinus Torvalds 32061da177e4SLinus Torvalds static __initdata unsigned long thash_entries; 32071da177e4SLinus Torvalds static int __init set_thash_entries(char *str) 32081da177e4SLinus Torvalds { 32091da177e4SLinus Torvalds if (!str) 32101da177e4SLinus Torvalds return 0; 32111da177e4SLinus Torvalds thash_entries = simple_strtoul(str, &str, 0); 32121da177e4SLinus Torvalds return 1; 32131da177e4SLinus Torvalds } 32141da177e4SLinus Torvalds __setup("thash_entries=", set_thash_entries); 32151da177e4SLinus Torvalds 32161da177e4SLinus Torvalds void __init tcp_init(void) 32171da177e4SLinus Torvalds { 32181da177e4SLinus Torvalds struct sk_buff *skb = NULL; 3219f03d78dbSEric Dumazet unsigned long limit; 3220c5ed63d6SEric Dumazet int i, max_share, cnt; 3221da5c78c8SWilliam Allen Simpson unsigned long jiffy = jiffies; 32221da177e4SLinus Torvalds 32231f9e636eSPavel Emelyanov BUILD_BUG_ON(sizeof(struct tcp_skb_cb) > sizeof(skb->cb)); 32241da177e4SLinus Torvalds 32251748376bSEric Dumazet percpu_counter_init(&tcp_sockets_allocated, 0); 3226dd24c001SEric Dumazet percpu_counter_init(&tcp_orphan_count, 0); 32276e04e021SArnaldo Carvalho de Melo tcp_hashinfo.bind_bucket_cachep = 32286e04e021SArnaldo Carvalho de Melo kmem_cache_create("tcp_bind_bucket", 32296e04e021SArnaldo Carvalho de Melo sizeof(struct inet_bind_bucket), 0, 323020c2df83SPaul Mundt SLAB_HWCACHE_ALIGN|SLAB_PANIC, NULL); 32311da177e4SLinus Torvalds 32321da177e4SLinus Torvalds /* Size and allocate the main established and bind bucket 32331da177e4SLinus Torvalds * hash tables. 32341da177e4SLinus Torvalds * 32351da177e4SLinus Torvalds * The methodology is similar to that of the buffer cache. 32361da177e4SLinus Torvalds */ 32376e04e021SArnaldo Carvalho de Melo tcp_hashinfo.ehash = 32381da177e4SLinus Torvalds alloc_large_system_hash("TCP established", 32390f7ff927SArnaldo Carvalho de Melo sizeof(struct inet_ehash_bucket), 32401da177e4SLinus Torvalds thash_entries, 32414481374cSJan Beulich (totalram_pages >= 128 * 1024) ? 324218955cfcSMike Stroyan 13 : 15, 32439e950efaSJohn Heffner 0, 32441da177e4SLinus Torvalds NULL, 3245f373b53bSEric Dumazet &tcp_hashinfo.ehash_mask, 32460ccfe618SJean Delvare thash_entries ? 0 : 512 * 1024); 3247f373b53bSEric Dumazet for (i = 0; i <= tcp_hashinfo.ehash_mask; i++) { 32483ab5aee7SEric Dumazet INIT_HLIST_NULLS_HEAD(&tcp_hashinfo.ehash[i].chain, i); 32493ab5aee7SEric Dumazet INIT_HLIST_NULLS_HEAD(&tcp_hashinfo.ehash[i].twchain, i); 32501da177e4SLinus Torvalds } 3251230140cfSEric Dumazet if (inet_ehash_locks_alloc(&tcp_hashinfo)) 3252230140cfSEric Dumazet panic("TCP: failed to alloc ehash_locks"); 32536e04e021SArnaldo Carvalho de Melo tcp_hashinfo.bhash = 32541da177e4SLinus Torvalds alloc_large_system_hash("TCP bind", 32550f7ff927SArnaldo Carvalho de Melo sizeof(struct inet_bind_hashbucket), 3256f373b53bSEric Dumazet tcp_hashinfo.ehash_mask + 1, 32574481374cSJan Beulich (totalram_pages >= 128 * 1024) ? 325818955cfcSMike Stroyan 13 : 15, 32599e950efaSJohn Heffner 0, 32606e04e021SArnaldo Carvalho de Melo &tcp_hashinfo.bhash_size, 32611da177e4SLinus Torvalds NULL, 32621da177e4SLinus Torvalds 64 * 1024); 32636e04e021SArnaldo Carvalho de Melo tcp_hashinfo.bhash_size = 1 << tcp_hashinfo.bhash_size; 32646e04e021SArnaldo Carvalho de Melo for (i = 0; i < tcp_hashinfo.bhash_size; i++) { 32656e04e021SArnaldo Carvalho de Melo spin_lock_init(&tcp_hashinfo.bhash[i].lock); 32666e04e021SArnaldo Carvalho de Melo INIT_HLIST_HEAD(&tcp_hashinfo.bhash[i].chain); 32671da177e4SLinus Torvalds } 32681da177e4SLinus Torvalds 3269c5ed63d6SEric Dumazet 3270c5ed63d6SEric Dumazet cnt = tcp_hashinfo.ehash_mask + 1; 3271c5ed63d6SEric Dumazet 3272c5ed63d6SEric Dumazet tcp_death_row.sysctl_max_tw_buckets = cnt / 2; 3273c5ed63d6SEric Dumazet sysctl_tcp_max_orphans = cnt / 2; 3274c5ed63d6SEric Dumazet sysctl_max_syn_backlog = max(128, cnt / 256); 32751da177e4SLinus Torvalds 3276f03d78dbSEric Dumazet limit = nr_free_buffer_pages() / 8; 327753cdcc04SJohn Heffner limit = max(limit, 128UL); 327853cdcc04SJohn Heffner sysctl_tcp_mem[0] = limit / 4 * 3; 327953cdcc04SJohn Heffner sysctl_tcp_mem[1] = limit; 328052bf376cSJohn Heffner sysctl_tcp_mem[2] = sysctl_tcp_mem[0] * 2; 32811da177e4SLinus Torvalds 328253cdcc04SJohn Heffner /* Set per-socket limits to no more than 1/128 the pressure threshold */ 32837b4f4b5eSJohn Heffner limit = ((unsigned long)sysctl_tcp_mem[1]) << (PAGE_SHIFT - 7); 32847b4f4b5eSJohn Heffner max_share = min(4UL*1024*1024, limit); 32857b4f4b5eSJohn Heffner 32863ab224beSHideo Aoki sysctl_tcp_wmem[0] = SK_MEM_QUANTUM; 32877b4f4b5eSJohn Heffner sysctl_tcp_wmem[1] = 16*1024; 32887b4f4b5eSJohn Heffner sysctl_tcp_wmem[2] = max(64*1024, max_share); 32897b4f4b5eSJohn Heffner 32903ab224beSHideo Aoki sysctl_tcp_rmem[0] = SK_MEM_QUANTUM; 32917b4f4b5eSJohn Heffner sysctl_tcp_rmem[1] = 87380; 32927b4f4b5eSJohn Heffner sysctl_tcp_rmem[2] = max(87380, max_share); 32931da177e4SLinus Torvalds 32941da177e4SLinus Torvalds printk(KERN_INFO "TCP: Hash tables configured " 3295f373b53bSEric Dumazet "(established %u bind %u)\n", 3296f373b53bSEric Dumazet tcp_hashinfo.ehash_mask + 1, tcp_hashinfo.bhash_size); 3297317a76f9SStephen Hemminger 3298317a76f9SStephen Hemminger tcp_register_congestion_control(&tcp_reno); 3299da5c78c8SWilliam Allen Simpson 3300da5c78c8SWilliam Allen Simpson memset(&tcp_secret_one.secrets[0], 0, sizeof(tcp_secret_one.secrets)); 3301da5c78c8SWilliam Allen Simpson memset(&tcp_secret_two.secrets[0], 0, sizeof(tcp_secret_two.secrets)); 3302da5c78c8SWilliam Allen Simpson tcp_secret_one.expires = jiffy; /* past due */ 3303da5c78c8SWilliam Allen Simpson tcp_secret_two.expires = jiffy; /* past due */ 3304da5c78c8SWilliam Allen Simpson tcp_secret_generating = &tcp_secret_one; 3305da5c78c8SWilliam Allen Simpson tcp_secret_primary = &tcp_secret_one; 3306da5c78c8SWilliam Allen Simpson tcp_secret_retiring = &tcp_secret_two; 3307da5c78c8SWilliam Allen Simpson tcp_secret_secondary = &tcp_secret_two; 33081da177e4SLinus Torvalds } 3309