nfscl: Fix handling of expired Kerberos credentials (NFSv4.1/4.2)

If the NFS server detects that the Kerberos credentials provided
by a NFSv4.1/4.2 mount using sec=krb5[ip] have expired, the NFS
ser

nfscl: Fix handling of expired Kerberos credentials (NFSv4.1/4.2)

If the NFS server detects that the Kerberos credentials provided
by a NFSv4.1/4.2 mount using sec=krb5[ip] have expired, the NFS
server replies with a krpc layer error of RPC_AUTHERROR.
When this happened, the client erroneously left the NFSv4.1/4.2
session slot busy, so that it could not be used by other RPCs.
If this happened for all session slots, the mount point would
hang.

This patch fixes the problem by releasing the session slot
and resetting its sequence# upon receiving a RPC_AUTHERROR
reply.

This bug only affects NFSv4.1/4.2 mounts using sec=krb5[ip],
but has existed since NFSv4.1 client support was added to
FreeBSD.

So, why has the bug remained undetected for so long?
I cannot be sure, but I suspect that, often, the client detected
the Kerberos credential expiration before attempting the RPC.
For this case, the client would not do the RPC and, as such,
there would be no busy session slot. Also, no hang would
occur until all session slots are busied (64 for a FreeBSD
client/server), so many cases of the bug probably went undetected?
Also, use of sec=krb5[ip] mounts are not that common.

PR: 275905
Tested by: Lexi <lexi.freebsd@le-fay.org>
MFC after: 1 week

show more ...

nfscl/kgssapi: Fix Kerberized NFS mounts to pNFS servers

During recent testing related to the IETF NFSv4 Bakeathon, it was
discovered that Kerberized NFSv4.1/4.2 mounts to pNFS servers
(sec=krb5[ip]

nfscl/kgssapi: Fix Kerberized NFS mounts to pNFS servers

During recent testing related to the IETF NFSv4 Bakeathon, it was
discovered that Kerberized NFSv4.1/4.2 mounts to pNFS servers
(sec=krb5[ip],pnfs mount options) was broken.
The FreeBSD client was using the "service principal" for
the MDS to try and establish a rpcsec_gss credential for a DS,
which is incorrect. (A "service principal" looks like
"nfs@<fqdn-of-server>" and the <fqdn-of-server> for the DS is not
the same as the MDS for most pNFS servers.)

To fix this, the rpcsec_gss code needs to be able to do a
reverse DNS lookup of the DS's IP address. A new kgssapi upcall
to the gssd(8) daemon is added by this patch to do the reverse DNS
along with a new rpcsec_gss function to generate the "service
principal".

A separate patch to the gssd(8) will be committed, so that this
patch will fix the problem. Without the gssd(8) patch, the new
upcall fails and current/incorrect behaviour remains.

This bug only affects the rare case of a Kerberized (sec=krb5[ip],pnfs)
mount using pNFS.

This patch changes the internal KAPI between the kgssapi and
nfscl modules, but since I did a version bump a few days ago,
I will not do one this time.

MFC after: 1 month

show more ...

nfscl: Handle the NFSERR_RETRYUNCACHEDREP error from a NFSv4 server

In a recent email list discussion related to NFSv4 mount problems
against a non-FreeBSD NFSv4 server, the reporter of the issue no

nfscl: Handle the NFSERR_RETRYUNCACHEDREP error from a NFSv4 server

In a recent email list discussion related to NFSv4 mount problems
against a non-FreeBSD NFSv4 server, the reporter of the issue noted
that the server had replied 10068 (NFSERR_RETRYUNCACHEDREP). This
did not seem related to the mount problem, but I had never seen this
error before. It indicates that an RPC retry after a new TCP
connection has been established failed because the server did not
cache the reply. Since this should only happen for idempotent
operations, redoing the RPC should be safe.

This patch modifies the NFSv4.1/4.2 client to redo the RPC instead
of considering the server error fatal. It should only affect the
unusual case where TCP connections to NFSv4 servers are breaking
without the NFSv4 server rebooting.

Reported by: J David <j.devid.lists@gmail.com>
MFC after: 2 weeks

show more ...

nfsd: Fix a server crash

PR#274346 reports a crash which appears to be caused by a NULL default session
being destroyed. This patch should avoid the crash.

Tested by: Joshua Kinard <freebsd@kumba.

nfsd: Fix a server crash

PR#274346 reports a crash which appears to be caused by a NULL default session
being destroyed. This patch should avoid the crash.

Tested by: Joshua Kinard <freebsd@kumba.dev>
PR: 274346
MFC after: 2 weeks

show more ...

sys: Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

nfscl: Add a new NFSv4.1/4.2 mount option for Kerberized mounts

Without this patch, a Kerberized NFSv4.1/4.2 mount must provide
a Kerberos credential for the client at mount time. This credential
i

nfscl: Add a new NFSv4.1/4.2 mount option for Kerberized mounts

Without this patch, a Kerberized NFSv4.1/4.2 mount must provide
a Kerberos credential for the client at mount time. This credential
is typically referred to as a "machine credential". It can be
created one of two ways:
- The user (usually root) has a valid TGT at the time the mount
is done and this becomes the machine credential.
There are two problems with this.
1 - The user doing the mount must have a valid TGT for a user
principal at mount time. As such, the mount cannot be put
in fstab(5) or similar.
2 - When the TGT expires, the mount breaks.
- The client machine has a service principal in its default keytab
file and this service principal (typically called a host-based
initiator credential) is used as the machine credential.
There are problems with this approach as well:
1 - There is a certain amount of administrative overhead creating
the service principal for the NFS client, creating a keytab
entry for this principal and then copying the keytab entry
into the client's default keytab file via some secure means.
2 - The NFS client must have a fixed, well known, DNS name, since
that FQDN is in the service principal name as the instance.

This patch uses a feature of NFSv4.1/4.2 called SP4_NONE, which
allows the state maintenance operations to be performed by any
authentication mechanism, to do these operations via AUTH_SYS
instead of RPCSEC_GSS (Kerberos). As such, neither of the above
mechanisms is needed.

It is hoped that this option will encourage adoption of Kerberized
NFS mounts using TLS, to provide a more secure NFS mount.

This new NFSv4.1/4.2 mount option, called "syskrb5" must be used
with "sec=krb5[ip]" to avoid the need for either of the above
Kerberos setups to be done by the client.

Note that all file access/modification operations still require
users on the NFS client to have a valid TGT recognized by the
NFSv4.1/4.2 server. As such, this option allows, at most, a
malicious client to do some sort of DOS attack.

Although not required, use of "tls" with this new option is
encouraged, since it provides on-the-wire encryption plus,
optionally, client identity verification via a X.509
certificate provided to the server during TLS handshake.
Alternately, "sec=krb5p" does provide on-the-wire
encryption of file data.

A mount_nfs(8) man page update will be done in a separate commit.

Discussed on: freebsd-current@
MFC after: 3 months

show more ...

nfscommon: Revert use of nfsstatsv1_p in nfs_commonkrpc.c

Commit 9d329bbc9aea converted a lot of accesses to nfsstatsv1
to use nfsstatsv1_p instead. However, the accesses in
nfs_commonkrpc.c are fo

nfscommon: Revert use of nfsstatsv1_p in nfs_commonkrpc.c

Commit 9d329bbc9aea converted a lot of accesses to nfsstatsv1
to use nfsstatsv1_p instead. However, the accesses in
nfs_commonkrpc.c are for client side and should not be
converted. This patch puts them back in the correct
pre-commit 9d329bbc9aea form.

MFC after: 3 months

show more ...

nfsd: Wrap nfsstatsv1_p in the NFSD_VNET() macro

Commit 7344856e3a6d added a lot of macros that will front end
vnet macros so that nfsd(8) can run in vnet prison.
The nfsstatsv1_p variable got misse

nfsd: Wrap nfsstatsv1_p in the NFSD_VNET() macro

Commit 7344856e3a6d added a lot of macros that will front end
vnet macros so that nfsd(8) can run in vnet prison.
The nfsstatsv1_p variable got missed. This patch wraps all
uses of nfsstatsv1_p with the NFSD_VNET() macro.
The NFSD_VNET() macro is still a null macro.

MFC after: 3 months

show more ...

nfsd: Continue adding macros so nfsd can run in a vnet prison

Commit 7344856e3a6d added a lot of macros that will front end
vnet macros so that nfsd(8) can run in vnet prison.
This patch adds some m

nfsd: Continue adding macros so nfsd can run in a vnet prison

Commit 7344856e3a6d added a lot of macros that will front end
vnet macros so that nfsd(8) can run in vnet prison.
This patch adds some more of them and also a lot of uses of
nfsstatsv1_p instead of nfsstatsv1. nfsstatsv1_p points to
nfsstatsv1 for prison0, but will point to a malloc'd structure
for other prisons.

It also puts nfsstatsv1_p in nfscommon.ko instead of nfsd.ko.

MFC after: 3 months

show more ...

nfscl: Add a console message for session recovery

The NFSv4.1/4.2 client does recovery when it receives a
NFSERR_BADSESSION reply from the server. If the server has
not rebooted, this is often caus

nfscl: Add a console message for session recovery

The NFSv4.1/4.2 client does recovery when it receives a
NFSERR_BADSESSION reply from the server. If the server has
not rebooted, this is often caused by multiple clients using
the same /etc/hostid and, as such, not being recognized as
different clients by the server.

This trivial patch adds a console message to suggest that
client's /etc/hostid's need to be checked for uniqueness.

MFC after: 2 weeks

show more ...

nfscl: Fix handling of nd_slotid while handling NFSERR_BADSESSION

When the NFSv4.1/4.2 client is handling a server error
of NFSERR_BADSESSION, it retries RPCs with a new session.
Without this patch,

nfscl: Fix handling of nd_slotid while handling NFSERR_BADSESSION

When the NFSv4.1/4.2 client is handling a server error
of NFSERR_BADSESSION, it retries RPCs with a new session.
Without this patch, the nd_slotid was not being updated
for the new session.

This would result in a bogus console message like
"Wrong session srvslot=X slot=Y" and then it would
free the incorrect slot, often generating a
"freeing free slot!!" console message as well.

This patch fixes the problem.

Note that FreeBSD NFSv4.1/4.2 servers only
generate a NFSERR_BADSESSION error after a reboot
or after a client does a DestroySession operation.

PR: 260011
MFC after: 1 week

show more ...

Revert "nfscl: Fix handling of nd_slotid while handling NFSERR_BADSESSION"

Revert this commit, since I now have a better fix to commit.

This reverts commit 8e59ec29e47f6ec64f54ddd88cab388ae536f0ff.

nfscl: Fix handling of nd_slotid while handling NFSERR_BADSESSION

When the NFSv4.1/4.2 client is handling a server error
of NFSERR_BADSESSION, it retries RPCs with a new session.
Without this patch,

nfscl: Fix handling of nd_slotid while handling NFSERR_BADSESSION

When the NFSv4.1/4.2 client is handling a server error
of NFSERR_BADSESSION, it retries RPCs with a new session.
Without this patch, the nd_slotid was not being updated
for the new session.

This would result in a bogus console message like
"Wrong session srvslot=X slot=Y" and then it would
free the incorrect slot, often generating a
"freeing free slot!!" console message as well.

This patch fixes the problem.

Note that FreeBSD NFSv4.1/4.2 servers only
generate a NFSERR_BADSESSION error after a reboot
or after a client does a DestroySession operation.

PR: 260011
MFC after: 1 week

show more ...

nfscl: Fix handling of a bad session slot (NFSv4.1/4.2)

When a session has been marked defunct by the server
sending a NFSERR_BADSESSION reply to the NFSv4.1/4.2
client, nfsv4_sequencelookup() retur

nfscl: Fix handling of a bad session slot (NFSv4.1/4.2)

When a session has been marked defunct by the server
sending a NFSERR_BADSESSION reply to the NFSv4.1/4.2
client, nfsv4_sequencelookup() returns NFSERR_BADSESSION
without actually assigning a session slot.
Without this patch, newnfs_request() would erroneously
free slot 0.

This could result in the slot being reused prematurely,
but most likely just generated a "freeing free slot!!"
console message.

This patch fixes the code to not do the erroneous
freeing of the slot for this case.

PR: 260011
MFC after: 1 week

show more ...

nfscl: Enable detection of bad session slots

To deal with broken session slots caused by the use of the
"soft" and/or "intr" mount options, nfsv4_sequencelookup()
has been modified to track the pote

nfscl: Enable detection of bad session slots

To deal with broken session slots caused by the use of the
"soft" and/or "intr" mount options, nfsv4_sequencelookup()
has been modified to track the potentially broken session
slots (commit 40ada74ee1da). Then, when all session slots
are potentially broken, nfsv4_sequencelookup() does a
DeleteSession operation, so that the NFSv4.1/4.2 server will
reply NFSERR_BADSESSION to uses of the session.
The client will then recover by doing a CreateSession to
acquire a new session.

This patch adds the code that marks potentially bad
slots, so that the above semantics become functional.
It has been successfully tested against a FreeBSD
NFSv4.1/4.2 server, but does not work against a Linux 5.15
NFSv4.1/4.2 server. (The Linux 5.15 server creates
a new session with the same sessionid as the destroyed
one and, as such, keeps returning NFSERR_BADSESSION.
I believe this is a bug in the Linux server.)

However, this should not cause a regression and will
make "intr" mounts fairly usable against the NFSv4.1/4.2
servers where it works.

PR: 260011
MFC after: 2 weeks

show more ...

nfscl: Add optional support for slots marked bad

This patch adds support for session slots marked bad
to nfsv4_sequencelookup(). An additional boolean
argument indicates if the check for slots mark

nfscl: Add optional support for slots marked bad

This patch adds support for session slots marked bad
to nfsv4_sequencelookup(). An additional boolean
argument indicates if the check for slots marked bad
should be done.

The "cred" argument added to nfscl_reqstart() by
commit 326bcf9394c7 is now passed into nfsv4_setquence()
so that it can optionally set the boolean argument
for nfsv4_sequencelookup(). When optionally enabled,
nfsv4_setsequence() will do a DestroySession when all
slots are marked bad.

Since the code that marks slots bad is not yet committed,
this patch should not result in a semantics change.

PR: 260011
MFC after: 2 weeks

show more ...

nfscl: Sanity check the Sequence slotid in reply

The slotid in the Sequence reply must be the same as
in the request. Check that it is the same and log
a console message if it is not, plus set it t

nfscl: Sanity check the Sequence slotid in reply

The slotid in the Sequence reply must be the same as
in the request. Check that it is the same and log
a console message if it is not, plus set it to the
correct value.

Reported by: rtm@lcs.mit.edu
Tested by: rtm@lcs.mit.edu
PR: 260071
MFC after: 2 weeks

show more ...

nfscl: Fix NFSv4.1/4.2 pnfs mounts using nconnect

When a mount with the "pnfs" and "nconnect" options specified
does an I/O operation, it erroneously uses a TCP connection
to the MDS when it is mean

nfscl: Fix NFSv4.1/4.2 pnfs mounts using nconnect

When a mount with the "pnfs" and "nconnect" options specified
does an I/O operation, it erroneously uses a TCP connection
to the MDS when it is meant to be a DS operation and, as such,
needs to use a TCP connection to the DS. This patch fixes this.

When the "pnfs" and "nconnect" options are specified for a
NFSv4.1/4.2 mount, there probably should be N connections
established to each DS for I/O RPCs. This is a fair amount
of work and may be done in a future commit.

This problem was found during a recent IETF NFSv4 working
group testing event.

MFC after: 2 weeks

show more ...

nfscl: Fix forced dismount when "nconnect" is specified

When a forced dismount is done and the "nconnect" mount
option was used, the additional connections must be closed.
This patch does that.

Fou

nfscl: Fix forced dismount when "nconnect" is specified

When a forced dismount is done and the "nconnect" mount
option was used, the additional connections must be closed.
This patch does that.

Found during a recent IETF NFSv4 working group testing event.

MFC after: 2 weeks

show more ...

nfscl: Use a smaller initial delay time for NFSERR_DELAY

For NFS RPCs that receive a NFSERR_DELAY reply, the delay time
is initially 1sec and then increases exponentially to NFS_TRYLATERDEL.
It was

nfscl: Use a smaller initial delay time for NFSERR_DELAY

For NFS RPCs that receive a NFSERR_DELAY reply, the delay time
is initially 1sec and then increases exponentially to NFS_TRYLATERDEL.
It was found that this delay time is excessive for some NFSv4
servers, which work well with a 1msec delay.
A 1sec delay resulted in very slow performance for Remove and
Rename when delegations and pNFS were enabled.

This patch decreases the initial delay time to 1msec.

Found during a recent IETF NFSv4 working group testing event.

MFC after: 2 weeks

show more ...

nfscl: Add a Linux compatible "nconnect" mount option

Linux has had an "nconnect" NFS mount option for some time.
It specifies that N (up to 16) TCP connections are to created for a mount,
instead o

nfscl: Add a Linux compatible "nconnect" mount option

Linux has had an "nconnect" NFS mount option for some time.
It specifies that N (up to 16) TCP connections are to created for a mount,
instead of just one TCP connection.

A discussion on freebsd-net@ indicated that this could improve
client<-->server network bandwidth, if either the client or server
have one of the following:
- multiple network ports aggregated to-gether with lagg/lacp.
- a fast NIC that is using multiple queues
It does result in using more IP port#s and might increase server
peak load for a client.

One difference from the Linux implementation is that this implementation
uses the first TCP connection for all RPCs composed of small messages
and uses the additional TCP connections for RPCs that normally have
large messages (Read/Readdir/Write). The Linux implementation spreads
all RPCs across all TCP connections in a round robin fashion, whereas
this implementation spreads Read/Readdir/Write across the additional
TCP connections in a round robin fashion.

Reviewed by: markj
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D30970

show more ...

nfscl: Improve "Consider increasing kern.ipc.maxsockbuf" message

When the setting of kern.ipc.maxsockbuf is less than what is
desired for I/O based on vfs.maxbcachebuf and vfs.nfs.bufpackets,
a cons

nfscl: Improve "Consider increasing kern.ipc.maxsockbuf" message

When the setting of kern.ipc.maxsockbuf is less than what is
desired for I/O based on vfs.maxbcachebuf and vfs.nfs.bufpackets,
a console message of "Consider increasing kern.ipc.maxsockbuf".
is printed.

This patch modifies the message to provide a suggested value
for kern.ipc.maxsockbuf.
Note that the setting is only needed when the NFS rsize/wsize
is set to vfs.maxbcachebuf.

While here, make nfs_bufpackets global, so that it can be used
by a future patch that adds a sysctl to set the NFS server's
maximum I/O size. Also, remove "sizeof(u_int32_t)" from the maximum
packet length, since NFS_MAXXDR is already an "overestimate"
of the actual length.

MFC after: 2 weeks

show more ...

nfsd: Reduce the callback timeout to 800msec

Recent discussion on the nfsv4@ietf.org mailing list confirmed
that an NFSv4 server should reply to an RPC in less than 1second.
If an NFSv4 RPC requires

nfsd: Reduce the callback timeout to 800msec

Recent discussion on the nfsv4@ietf.org mailing list confirmed
that an NFSv4 server should reply to an RPC in less than 1second.
If an NFSv4 RPC requires a delegation be recalled,
the server will attempt a CB_RECALL callback.
If the client is not responsive, the RPC reply will be delayed
until the callback times out.
Without this patch, the timeout is set to 4 seconds (set in
ticks, but used as seconds), resulting in the RPC reply taking over 4sec.
This patch redefines the constant as being in milliseconds and it
implements that for a value of 800msec, to ensure the RPC
reply is sent in less than 1second.

This patch only affects mounts from clients when delegations
are enabled on the server and the client is unresponsive to callbacks.

MFC after: 2 weeks

show more ...

nfscl: fix the handling of NFSERR_DELAY for Open/LayoutGet RPCs

For a pNFS mount, the NFSv4.1/4.2 client uses compound RPCs that
have both Open and LayoutGet operations in them.
If the pNFS server w

nfscl: fix the handling of NFSERR_DELAY for Open/LayoutGet RPCs

For a pNFS mount, the NFSv4.1/4.2 client uses compound RPCs that
have both Open and LayoutGet operations in them.
If the pNFS server were tp reply NFSERR_DELAY for one of these
compounds, the retry after a delay cannot be handled by
newnfs_request(), since there is a reference held on the open
state for the Open operation in them.

Fix this by adding these RPCs to the "don't do delay here"
list in newnfs_request().

This patch is only needed if the mount is using pNFS (the "pnfs"
mount option) and probably only matters if the MDS server
is issuing delegations as well as pNFS layouts.

Found by code inspection.

MFC after: 2 weeks

show more ...

nfsd: fix the slot sequence# when a callback fails

Commit 4281bfec3628 patched the server so that the
callback session slot would be free'd for reuse when
a callback attempt fails.
However, this can

nfsd: fix the slot sequence# when a callback fails

Commit 4281bfec3628 patched the server so that the
callback session slot would be free'd for reuse when
a callback attempt fails.
However, this can often result in the sequence# for
the session slot to be advanced such that the client
end will reply NFSERR_SEQMISORDERED.

To avoid the NFSERR_SEQMISORDERED client reply,
this patch negates the sequence# advance for the
case where the callback has failed.
The common case is a failed back channel, where
the callback cannot be sent to the client, and
not advancing the sequence# is correct for this
case. For the uncommon case where the client's
reply to the callback is lost, not advancing the
sequence# will indicate to the client that the
next callback is a retry and not a new callback.
But, since the FreeBSD server always sets "csa_cachethis"
false in the callback sequence operation, a retry
and a new callback should be handled the same way
by the client, so this should not matter.

Until you have this patch in your NFSv4.1/4.2 server,
you should consider avoiding the use of delegations.
Even with this patch, interoperation with the
Linux NFSv4.1/4.2 client in kernel versions prior
to 5.3 can result in frequent 15second delays if
delegations are enabled. This occurs because, for
kernels prior to 5.3, the Linux client does a TCP
reconnect every time it sees multiple concurrent
callbacks and then it takes 15seconds to recover
the back channel after doing so.

MFC after: 2 weeks

show more ...