kernel: Fix defining of .init_array and .fini_array sections

These input sections can have decimal numbers as the priority suffix.
Clang emits the '%u' form, while SORT is an alias for SORT_BY_NAME,

kernel: Fix defining of .init_array and .fini_array sections

These input sections can have decimal numbers as the priority suffix.
Clang emits the '%u' form, while SORT is an alias for SORT_BY_NAME,
hence will result in wrong order of constructors / destructors in
output sections. Fix by using the correct sorting command
SORT_BY_INIT_PRIORITY instead [1].

The functions referenced by section .fini_array is in the normal order,
but been executed in the reverse order. The order is same with
.init_array section.

Currently these sections are not used, there should be no functional
change.

Note: As for the .ctors and .dtors sections, both Clang and GCC emit
the priority suffix in the form of '%05u', so there is no semantic
difference between SORT_BY_NAME and SORT_BY_INIT_PRIORITY for those
sections [2].

This fix is extracted from a bigger patch [3] of hselasky, with
additional fix for .fini_array section.

1. https://sourceware.org/binutils/docs/ld/Input-Section-Wildcards.html
2. https://reviews.llvm.org/D91187
3. https://reviews.freebsd.org/D40467

Reviewed by: imp (previous version)
Obtained from: hselasky
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D45194

show more ...

kernel: Move the debug stuff into a common script

Move a copy of amd64's debug code into debug.ldscript. Make all the
kernels use this. This has the effect of modernizing the STABS for
powerpc as th

kernel: Move the debug stuff into a common script

Move a copy of amd64's debug code into debug.ldscript. Make all the
kernels use this. This has the effect of modernizing the STABS for
powerpc as the others were almost already in sync. For the ones that
weren't this adds the DWARF 3 debug symbols from i386/amd64.

Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D44071

show more ...

kernel: Sync ldscript files wrt gnu.attributes and .note.GNU-stack

Bring the keeping of .gnu.attributes to all architectures. Also discard
.note.GNU-stack on all archtiectures. Plus delete obsolete

kernel: Sync ldscript files wrt gnu.attributes and .note.GNU-stack

Bring the keeping of .gnu.attributes to all architectures. Also discard
.note.GNU-stack on all archtiectures. Plus delete obsolete comment that
was removed from i386 in 2010.

Sponsored by: Netflix
Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D44069

show more ...

sys: Remove $FreeBSD$: one-line .c comment pattern

Remove /^/[*/]\s*\$FreeBSD\$.*\n/

amd64: do not assume that kernel is loaded at 2M physical

Allow any 2M aligned contiguous location below 4G for the staging
area location. It should still be mapped by loader at KERNBASE.

The assu

amd64: do not assume that kernel is loaded at 2M physical

Allow any 2M aligned contiguous location below 4G for the staging
area location. It should still be mapped by loader at KERNBASE.

The assumption kernel makes about loader->kernel handoff with regard to
the MMU programming are explicitly listed at the beginning of hammer_time(),
where kernphys is calculated. Now kernphys is the variable instead of
symbol designating the physical address.

Reviewed by: markj
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D31121

show more ...

Unobfuscate "KERNLOAD" parameter on amd64. This change lines-up amd64 with the
i386 and the rest of supported architectures by defining KERNLOAD in the
vmparam.h and getting rid of magic constant in

Unobfuscate "KERNLOAD" parameter on amd64. This change lines-up amd64 with the
i386 and the rest of supported architectures by defining KERNLOAD in the
vmparam.h and getting rid of magic constant in the linker script, which albeit
documented via comment but isn't programmatically accessible at a compile time.

Use KERNLOAD to eliminate another (matching) magic constant 100 lines down
inside unremarkable TU "copy.c" 3 levels deep in the EFI loader tree.

Reviewed by: markj
Approved by: markj
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D27355

show more ...

Tighten mapping protections on preloaded files on amd64.

- We load the kernel at 0x200000. Memory below that address need not
be executable, so do not map it as such.
- Remove references to .ldat

Tighten mapping protections on preloaded files on amd64.

- We load the kernel at 0x200000. Memory below that address need not
be executable, so do not map it as such.
- Remove references to .ldata and related sections in the kernel linker
script. They come from ld.bfd's default linker script, but are not
used, and we now use ld.lld to link the amd64 kernel. lld does not
contain a default linker script.
- Pad the .bss to a 2MB as we do between .text and .data. This
forces the loader to load additional files starting in the following
2MB page, preserving the use of superpage mappings for kernel data.
- Map memory above the kernel image with NX. The kernel linker now
upgrades protections as needed, and other preloaded file types
(e.g., entropy, microcode) need not be mapped with execute permissions
in the first place.

Reviewed by: kib
MFC after: 1 month
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D21859

show more ...

MFHead @348740

Sponsored by: The FreeBSD Foundation

Expose the kernel's build-ID through sysctl

After our migration (of certain architectures) to lld the kernel is built
with a unique build-ID. Make it available via a sysctl and uname(1) to
allow th

Expose the kernel's build-ID through sysctl

After our migration (of certain architectures) to lld the kernel is built
with a unique build-ID. Make it available via a sysctl and uname(1) to
allow the user to identify their running kernel.

Submitted by: Ali Mashtizadeh <ali_mashtizadeh.com>
MFC after: 2 weeks
Relnotes: Yes
Event: Waterloo Hackathon 2019
Differential Revision: https://reviews.freebsd.org/D20326

show more ...

Merge ^/head r340235 through r340367.

Add comment to explain kernel ldscript 0x200000 constant

Reported by: linimon

amd64: tweak the read_frequently section

1. align to 128 bytes to avoid possible waste from the preceeding section
2. sort entries by alignment SORT_BY_ALIGNMENT, plugging the holes (most
entries ar

amd64: tweak the read_frequently section

1. align to 128 bytes to avoid possible waste from the preceeding section
2. sort entries by alignment SORT_BY_ALIGNMENT, plugging the holes (most
entries are one byte in size, but they got interleaved with bigger ones)

Interestingly I was looking for a feature of the sort earlier and failed
to find it. It turns out the script was already utilizing sorting in other
places, so shame on me.

Thanks for Travis Geiselbrecht for pointing me at the feature.

show more ...

amd64: align the .data.exclusive_cache_line section to 128

This aligns the section itself compared to other sections, does not change
internal alignment of fields stored inside. This may or may not

amd64: align the .data.exclusive_cache_line section to 128

This aligns the section itself compared to other sections, does not change
internal alignment of fields stored inside. This may or may not come later.

The motivation is partially combating adverse effects of the adjacent cache
line prefetcher. Without the annotation part of read_mostly section was on
the line of fire.

show more ...

amd64: Protect the kernel text, data, and BSS by setting the RW/NX bits
correctly for the data contained on each memory page.

There are several components to this change:
* Add a variable to indica

amd64: Protect the kernel text, data, and BSS by setting the RW/NX bits
correctly for the data contained on each memory page.

There are several components to this change:
* Add a variable to indicate the start of the R/W portion of the
initial memory.
* Stop detecting NX bit support for each AP. Instead, use the value
from the BSP and, if supported, activate the feature on the other
APs just before loading the correct page table. (Functionally, we
already assume that the BSP and all APs had the same support or
lack of support for the NX bit.)
* Set the RW and NX bits correctly for the kernel text, data, and
BSS (subject to some caveats below).
* Ensure DDB can write to memory when necessary (such as to set a
breakpoint).
* Ensure GDB can write to memory when necessary (such as to set a
breakpoint). For this purpose, add new MD functions gdb_begin_write()
and gdb_end_write() which the GDB support code can call before and
after writing to memory.

This change is not comprehensive:
* It doesn't do anything to protect modules.
* It doesn't do anything for kernel memory allocated after the kernel
starts running.
* In order to avoid excessive memory inefficiency, it may let multiple
types of data share a 2M page, and assigns the most permissions
needed for data on that page.

Reviewed by: jhb, kib
Discussed with: emaste
MFC after: 2 weeks
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D14282

show more ...

MFH @ r323558.

MFhead@r323343

Introduce __read_frequently

While __read_mostly groups variables together, their placement is not
specified. In particular 2 frequently used variables can end up in
different lines.

This annotation

Introduce __read_frequently

While __read_mostly groups variables together, their placement is not
specified. In particular 2 frequently used variables can end up in
different lines.

This annotation is only expected to be used for variables read all the time,
e.g. on each syscall entry.

MFC after: 1 week

show more ...

use INT3 instead of NOP for x86 binary padding

We should never end up executing the inter-function padding, so we
are better off faulting than silently carrying on to whatever function
happens to be

use INT3 instead of NOP for x86 binary padding

We should never end up executing the inter-function padding, so we
are better off faulting than silently carrying on to whatever function
happens to be next.

Note that LLD will soon do this by default (although it currently pads
with zeros).

Reviewed by: dim, kib
MFC after: 1 month
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D10047

show more ...

MFhead@r313243

Merge ^/head r312720 through r312893.

Introduce __read_mostly and __exclusive_cache_line macros.

The intended use is to annotate frequently used globals which either rarely
change (and thus can be grouped in the same cacheline) or are a

Introduce __read_mostly and __exclusive_cache_line macros.

The intended use is to annotate frequently used globals which either rarely
change (and thus can be grouped in the same cacheline) or are an atomic counter
(which means it may benefit from being the only variable in the cacheline).

Linker script support is provided only for amd64. Architectures without it risk
having other variables put in, i.e. as if they were not annotated. This is
harmless from correctness point of view.

Reviewed by: bde (previous version)
MFC after: 1 month

show more ...

Use explicit 0x200000 instead of MAXPAGESIZE for the amd64 kernel physaddr

MAXPAGESIZE is not well defined by the GNU ld documentation.
Different linkers, and different versions of the same linker,

Use explicit 0x200000 instead of MAXPAGESIZE for the amd64 kernel physaddr

MAXPAGESIZE is not well defined by the GNU ld documentation.
Different linkers, and different versions of the same linker, use
different MAXPAGESIZE values. Current versions of GNU gold and LLVM's
lld use 4K. When set to 4K the kernel panics at boot due to an issue
with x86bios.

Here we want the kernel physaddr to be the amd64 superpage size, so use
that value (2MB) explicitly. With this change GNU gold and LLVM lld can
link a working amd64 kernel.

PR: 214718 (x86bios)
Differential Revision: https://reviews.freebsd.org/D8610

show more ...

Merge ^/head r303250 through r304235.

remove CONSTRUCTORS from kernel linker scripts

The linker script CONSTRUCTORS keyword is only meaningful "when linking
object file formats which do not support arbitrary sections, such as
ECOFF and

remove CONSTRUCTORS from kernel linker scripts

The linker script CONSTRUCTORS keyword is only meaningful "when linking
object file formats which do not support arbitrary sections, such as
ECOFF and XCOFF"[1] and is ignored for other object file formats.

LLVM's lld does not yet accept (and ignore) CONSTRUCTORS, so just remove
CONSTRUCTORS from the linker scripts as it has no effect.

[1] https://sourceware.org/binutils/docs/ld/Output-Section-Keywords.html

Reviewed by: kib
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D7343

show more ...

Merge from head