ipmi: remove timeout from the ipmi_driver_request method

Driver requests are done with stack allocated request. The request is
put on the tailq and then we msleep(9) until kernel process processes

ipmi: remove timeout from the ipmi_driver_request method

Driver requests are done with stack allocated request. The request is
put on the tailq and then we msleep(9) until kernel process processes it.
If we timeout from this sleep, the kernel process may still read the
request from our stack, which may already be reused by some other code.

Make this sleep unbound and rely on the kernel process that does all its
I/O with timouts and will eventually wake us up.

Reviewed by: jhb
Differential Revision: https://reviews.freebsd.org/D47179

show more ...

sys: Automated cleanup of cdefs and other formatting

Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

Remov

sys: Automated cleanup of cdefs and other formatting

Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

Remove /^#if.*\n#endif.*\n#include\s+<sys/cdefs.h>.*\n/
Remove /\n+#include\s+<sys/cdefs.h>.*\n+#if.*\n#endif.*\n+/
Remove /\n+#if.*\n#endif.*\n+/
Remove /^#if.*\n#endif.*\n/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/types.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/param.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/capsicum.h>/

Sponsored by: Netflix

show more ...

sys: Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

spdx: The BSD-2-Clause-FreeBSD identifier is obsolete, drop -FreeBSD

The SPDX folks have obsoleted the BSD-2-Clause-FreeBSD identifier. Catch
up to that fact and revert to their recommended match of

spdx: The BSD-2-Clause-FreeBSD identifier is obsolete, drop -FreeBSD

The SPDX folks have obsoleted the BSD-2-Clause-FreeBSD identifier. Catch
up to that fact and revert to their recommended match of BSD-2-Clause.

Discussed with: pfg
MFC After: 3 days
Sponsored by: Netflix

show more ...

ipmi: use a queue for kcs driver requests when possible

The ipmi watchdog pretimeout action can trigger unintentionally in
certain rare, complicated situations. What we have seen at Netflix
is that

ipmi: use a queue for kcs driver requests when possible

The ipmi watchdog pretimeout action can trigger unintentionally in
certain rare, complicated situations. What we have seen at Netflix
is that the BMC can sometimes be sent a continuous stream of
writes to port 0x80, and due to what is a bug or misconfiguration
in the BMC software, this results in the BMC running out of memory,
becoming very slow to respond to KCS requests, and eventually being
rebooted by its own internal watchdog. While that is going on in
the BMC, back in the host OS, a number of requests are pending in
the ipmi request queue, and the kcs_loop thread is working on
processing these requests. All of the KCS accesses to process
those requests are timing out and eventually failing because the
BMC is responding very slowly or not at all, and the kcs_loop thread
is holding the IPMI_IO_LOCK the whole time that is going on.
Meanwhile the watchdogd process in the host is trying to pat the
BMC watchdog, and this process is sleeping waiting to get the
IPMI_IO_LOCK. It's not entirely clear why the watchdogd process
is sleeping for this lock, because the intention is that a thread
holding the IPMI_IO_LOCK should not sleep and thus any thread
that wants the lock should just spin to wait for it. My best guess
is that the kcs_loop thread is spinning waiting for the BMC to
respond for so long that it is eventually preempted, and during
the brief interval when the kcs_loop thread is not running,
the watchdogd thread notices that the lock holder is not running
and sleeps. When the kcs_loop thread eventually finishes processing
one request, it drops the IPMI_IO_LOCK and then immediately takes the
lock again so it can process the next request in the queue.
Because the watchdogd thread is sleeping at this point, the kcs_loop
always wins the race to acquire the IPMI_IO_LOCK, thus starving
the watchdogd thread. The callout for the watchdog pretimeout
would be reset by the watchdogd thread after its request to the BMC
watchdog completes, but since that request never processed, the
pretimeout callout eventually fires, even though there is nothing
actually wrong with the host.

To prevent this saga from unfolding:

- when kcs_driver_request() is called in a context where it can sleep,
queue the request and let the worker thread process it rather than
trying to process in the original thread.
- add a new high-priority queue for driver requests, so that the
watchdog patting requests will be processed as quickly as possible
even if lots of application requests have already been queued.

With these two changes, the watchdog pretimeout action does not trigger
even if the BMC is completely out to lunch for long periods of time
(as long as the watchdogd check command does not also get stuck).

Sponsored by: Netflix
Reviewed by: imp
Differential Revision: https://reviews.freebsd.org/D36555

show more ...

ipmi: Remove write only variables used to read form hardware

Sponsored by: Netflix

MFHead @349234

Sponsored by: The FreeBSD Foundation

The current IPMI KCS code is waiting 100us for all transitions (roughly
between each byte either sent or received). However, most transitions
actually complete in 2-3 microseconds.

By polling the st

The current IPMI KCS code is waiting 100us for all transitions (roughly
between each byte either sent or received). However, most transitions
actually complete in 2-3 microseconds.

By polling the status register with a delay of 4us with exponential
backoff, the performance of most IPMI operations is significantly
improved:
- A BMC update on a Supermicro x9 or x11 motherboard goes from ~1 hour
to ~6-8 minutes.
- An ipmitool sensor list time improves by a factor of 4.

Testing showed no significant improvements on a modern server by using
a lower delay.

The changes should also generally reduce the total amount of CPU or
I/O bandwidth used for a given IPMI operation.

Submitted by: Loic Prylli <lprylli@netflix.com>
Reviewed by: jhb
MFC after: 2 weeks
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D20527

show more ...

In cases where an application issues certain IPMI commands at a high
enough rate, the IPMI code can print large numbers of messages to the
console, such as:
ipmi0: KCS: Failed to read completion co

In cases where an application issues certain IPMI commands at a high
enough rate, the IPMI code can print large numbers of messages to the
console, such as:
ipmi0: KCS: Failed to read completion code
ipmi0: KCS error: ff
ipmi0: KCS: Failed to read completion code
ipmi0: KCS error: ff

These seem to be innocuous from a system standpoint, and the user-
space code can deal with the failures. Therefore, suppress printing
these messages to the console unless bootverbose is enabled.

Obtained from: Netflix, Inc.

show more ...

sys/dev: further adoption of SPDX licensing ID tags.

Mainly focus on files that use BSD 2-Clause license, however the tool I
was using misidentified many licenses so this was mostly a manual - error

sys/dev: further adoption of SPDX licensing ID tags.

Mainly focus on files that use BSD 2-Clause license, however the tool I
was using misidentified many licenses so this was mostly a manual - error
prone - task.

The Software Package Data Exchange (SPDX) group provides a specification
to make it easier for automated tools to detect and summarize well known
opensource licenses. We are gradually adopting the specification, noting
that the tags are considered only advisory and do not, in any way,
superceed or replace the license texts.

show more ...

Catch up with HEAD (r280229-r284686).

Merge sync of head

Merge from head

MFH: r281855-r282312

Sponsored by: The FreeBSD Foundation

Watchdog drivers need to support rearming the watchdog in contexts which
are not permitted to sleep. Only use the IPMI watchdog with backends
which poll driver-initiated requests to meet this requir

Watchdog drivers need to support rearming the watchdog in contexts which
are not permitted to sleep. Only use the IPMI watchdog with backends
which poll driver-initiated requests to meet this requirement.

In practice this means that watchdogs will no longer be used on systems
that use the SSIF backend.

Differential Revision: https://reviews.freebsd.org/D2062
MFC after: 2 weeks

show more ...

MFH: r278202,r278205-r278590

Sponsored by: The FreeBSD Foundation

Merge from HEAD

Merging ^/head r278298 through r278350.

Use direct hardware access for internal requests for KCS and SMIC. In
particular, updates to the watchdog should no longer sleep.
- Add a new IPMI_IO_LOCK for low-level I/O access. Use this for
k

Use direct hardware access for internal requests for KCS and SMIC. In
particular, updates to the watchdog should no longer sleep.
- Add a new IPMI_IO_LOCK for low-level I/O access. Use this for
kcs_polled_request() and smic_polled_request().
- Add a new backend callback "ipmi_driver_request" to handle a driver
request. The new callback performs the request sychronously for KCS
and SMIC. SSIF still defers the work to the worker thread since the
worker thread sleeps during request processing anyway.
- Allocate driver requests on the stack rather than using malloc().

Differential Revision: https://reviews.freebsd.org/D1723
Tested by: scottl
MFC after: 2 weeks

show more ...

Reintegrate head: r274132-r277384

Sponsored by: The FreeBSD Foundation

Reintegrate head revisions r273096-r277147

Sponsored by: The FreeBSD Foundation

MFhead @ r276594

Merge ^/head r274961 through r276301.

Explicitly treat timeouts when waiting for IBF or OBF to change state as an
error. This fixes occasional hangs in the IPMI kcs thread when using
ipmitool locally.

MFC after: 1 week

IFC @253398