release: Standardize on *-${FS}.* VM image naming

We publish three sets of VM images on download.freebsd.org:
* Generic VM images
* BASIC-CLOUDINIT VM images
* BASIC-CI VM images

Of these, the firs

release: Standardize on *-${FS}.* VM image naming

We publish three sets of VM images on download.freebsd.org:
* Generic VM images
* BASIC-CLOUDINIT VM images
* BASIC-CI VM images

Of these, the first had names like
FreeBSD-14.2-RELEASE-amd64-ufs.raw.xz,
the second had names like
FreeBSD-14.2-RELEASE-amd64-BASIC-CLOUDINIT.ufs.qcow2.xz,
and the third had names like
FreeBSD-14.2-RELEASE-amd64-BASIC-CI.ufs.raw.xz
but were listed in the CHECKSUM files with names like
FreeBSD-14.2-RELEASE-amd64-BASIC-CI-ufs.raw.xz

Standardize these to consistently use a hyphen before the filesystem
type rather than a period.

Note: On FreeBSD 14 we ship images without the -${FS} component in
their names; these are hardlinks to the -ufs images for compatibility
purposes. On FreeBSD 13 we only have UFS images and don't include the
filesystem in the name.

MFC after: 1 week
Reported by: jmg
Sponsored by: Amazon

show more ...

release: tweak Oracle Cloud settings

- use raw image disk type and enable zfs, this yields smaller
images for upload after using native qcow2 + zstd compression

Reviewed by: lwhsu, emaste
Differe

release: tweak Oracle Cloud settings

- use raw image disk type and enable zfs, this yields smaller
images for upload after using native qcow2 + zstd compression

Reviewed by: lwhsu, emaste
Differential Revision: https://reviews.freebsd.org/D47055
MFC after: 3 days
Approved by: emaste

show more ...

release: move OCI to ORACLE

This allows future releng tooling to use OCI for the industry
standard Open Container Initiative tooling, reducing potential
for confusion.

Approved by: cperciva
Reviewe

release: move OCI to ORACLE

This allows future releng tooling to use OCI for the industry
standard Open Container Initiative tooling, reducing potential
for confusion.

Approved by: cperciva
Reviewed by: emaste
Differential Revision: https://reviews.freebsd.org/D46975

MFC after: 3 days
Sponsored by: SkunkWerks, GmbH

show more ...

EC2: Add new "small" AMIs

These are the same as the standard "base" images except:

* They don't have kernel or world debug symbols,
* They don't have FreeBSD tests,
* They don't have 32-bit librari

EC2: Add new "small" AMIs

These are the same as the standard "base" images except:

* They don't have kernel or world debug symbols,
* They don't have FreeBSD tests,
* They don't have 32-bit libraries,
* They don't have LLDB,
* They don't have the Amazon SSM Agent pre-installed,
* They don't default to installing the awscli at first boot.

This reduces the amount of disk space in use when the EC2 instance
finishes booting from ~5 GB to ~1 GB.

Sponsored by: Amazon
Differential Revision: https://reviews.freebsd.org/D46509

show more ...

release: Bump default VM size for riscv64 to 6 GB

Due to issues with the riscv64 toolchain, some binaries end up
significantly larger on riscv64 than they should be. This results
in riscv64 VM imag

release: Bump default VM size for riscv64 to 6 GB

Due to issues with the riscv64 toolchain, some binaries end up
significantly larger on riscv64 than they should be. This results
in riscv64 VM images -- and at present *only* riscv64 images -- not
fitting within the default 5 GB filesystem size.

Bump the default size for riscv64 to 6 GB until the toolchain issues
can be resolved.

MFC after: 1 week
Sponsored by: Amazon

show more ...

release: Stop publishing non-FS-tagged VM images

VM, BASIC-CI, and BASIC-CLOUDWARE images are published with the
filesystem type (UFS or ZFS) in the file name. For backwards
compatibility we have p

release: Stop publishing non-FS-tagged VM images

VM, BASIC-CI, and BASIC-CLOUDWARE images are published with the
filesystem type (UFS or ZFS) in the file name. For backwards
compatibility we have published the UFS images with no filesystem
in the filename (aka with the same file names as we used before we
supported building ZFS VM images); those are going away now.

This will not be MFCed.

show more ...

release: Publish non-uploaded cloudware bits

While most "cloudware" (AWS, Azure, GCP, Vagrant) images get uploaded
to their respective clouds, we have two images -- BASIC-CI and
BASIC-CLOUDINIT -- w

release: Publish non-uploaded cloudware bits

While most "cloudware" (AWS, Azure, GCP, Vagrant) images get uploaded
to their respective clouds, we have two images -- BASIC-CI and
BASIC-CLOUDINIT -- which are published via the FTP mirrors. We have
been handling these using a manual and error-prone process of copying
them from the internal build directories into the FTP staging tree,
but they should be handled using the "make install ftp-stage"
mechanism which is used for all the other published images.

This commit 1. Adds a new 'make cloud-install' target which is run as
part of 'make install' and installs BASIC-CI and BASIC-CLOUDINIT images,
and 2. Extends 'make ftp-stage' in Makefile.mirrors to copy those bits
into the FTP staging area.

As a side benefit, this will also make BASIC-CI and BASIC-CLOUDINIT
images available for arm64; until now they have only been available
for amd64.

MFC after: 1 week

show more ...

Revert "release/Makefile.vm: Use qemu-user-static-devel"

This breaks the weekly snapshot builds. We'll try again once Warner has
some time to debug the qemu-user-static-devel port.

This reverts co

Revert "release/Makefile.vm: Use qemu-user-static-devel"

This breaks the weekly snapshot builds. We'll try again once Warner has
some time to debug the qemu-user-static-devel port.

This reverts commit a04ecddfc721427fd97fd1fcc27492ebd19ad183.

show more ...

release/Makefile.vm: Use qemu-user-static-devel

This is a newer version of the qemu-user-static code, which is very
much out of date.

Requested by: imp

release: make dependencies, not recursive make

For historical reasons, the vm-release and cloudware-release targets
were written as recursive makes of ${VMTARGETS} and ${CLOUDTARGETS};
this worked f

release: make dependencies, not recursive make

For historical reasons, the vm-release and cloudware-release targets
were written as recursive makes of ${VMTARGETS} and ${CLOUDTARGETS};
this worked fine until we started running `make release -jN` and
had both VM and cloud targets depending on ${QEMUTGT}, at which
point things exploded when we tried to build the port twice at the
same time.

Switch vm-release and cloudware-release to have make dependencies on
their individual components; this way a single make process runs and
is aware of the duplicate dependency (and only runs it once).

MFC after: 5 days

show more ...

release: Use qemu when cross-building vm images

For a bit over 5 years, we have used qemu when cross-building cloudware
images; in particular, it's necessary when installing packages which
might inc

release: Use qemu when cross-building vm images

For a bit over 5 years, we have used qemu when cross-building cloudware
images; in particular, it's necessary when installing packages which
might include post-install scripts.

Use qemu in the vm-images target too; while "generic" vm images don't
install packages, they still run newaliases and /etc/rc.d/ldconfig,
both of which fail without appropriate emulation.

MFC after: 1 week

show more ...

Makefile.vm: Set CLEANDIRS correctly for vm-image

With multiple filesystems and disk image formats, we now use different
staging directories; we need to include them all in CLEANDIRS.

Fixes: 97bd53

Makefile.vm: Set CLEANDIRS correctly for vm-image

With multiple filesystems and disk image formats, we now use different
staging directories; we need to include them all in CLEANDIRS.

Fixes: 97bd53ef4d20b "Makefile.vm: Fix duplicate rc.conf files"
MFC after: 3 days

show more ...

cloudware: allow disk format to be a list

Make basic-cloudinit available both in qcow2 and raw formats

MFC After: 3 days
Co-Authored-By: bapt
Sponsored by: OVHCloud (original patch from bapt)
Diffe

cloudware: allow disk format to be a list

Make basic-cloudinit available both in qcow2 and raw formats

MFC After: 3 days
Co-Authored-By: bapt
Sponsored by: OVHCloud (original patch from bapt)
Differential Revision: https://reviews.freebsd.org/D44747

show more ...

release/Makefile.vm: Support read-only ports tree

Build qemu (if needed) with WRKDIRPREFIX=/tmp/ports DISTDIR=/tmp/distfiles
so that we can have a read-only /usr/ports and don't contaminate it. Thi

release/Makefile.vm: Support read-only ports tree

Build qemu (if needed) with WRKDIRPREFIX=/tmp/ports DISTDIR=/tmp/distfiles
so that we can have a read-only /usr/ports and don't contaminate it. This
became an issue when I enabled parallel release building, since one image
might be creating its ports.txz file at the same time as we're building
qemu as a prerequisite for building another image.

MFC after: 5 days

show more ...

Revert "cloudware: allow disk format to be a list"

This reverts commit 6ec9aaf63c81a68881cb6312f777349a0ac82ad5.

Requested by: cperciva

cloudware: allow disk format to be a list

Make basic-cloudinit available both in qcow2 and raw formats

MFC After: 1 week
Reviewed by: Allanjude
Sponsored by: OVHCloud
Differential Revision: https:/

cloudware: allow disk format to be a list

Make basic-cloudinit available both in qcow2 and raw formats

MFC After: 1 week
Reviewed by: Allanjude
Sponsored by: OVHCloud
Differential Revision: https://reviews.freebsd.org/D44747

show more ...

cloudinit: fix creation of cloudinit ready VMs

The framework do not support multiple format images, so this last minute
addition of qcow2 format was clearly not a good idea.

while here fix a typo

cloudinit: fix creation of cloudinit ready VMs

The framework do not support multiple format images, so this last minute
addition of qcow2 format was clearly not a good idea.

while here fix a typo

Reported by: cperciva

show more ...

release: add basic cloudinit images

Provide both zfs and ufs images which a 1MB partition reserved for the
config drive wearing a GPT Label "config-drive" to allow consumer to
know where they should

release: add basic cloudinit images

Provide both zfs and ufs images which a 1MB partition reserved for the
config drive wearing a GPT Label "config-drive" to allow consumer to
know where they should push the config drive on the provided image.

2 formats available: qcow2 and raw

This has been tested on OVHCloud baremetal via "bring your own image"
Also tested on openstack

Reviewed by: emaste
Sponsored by: OVHCloud
Differential Revision: https://reviews.freebsd.org/D44369

show more ...

release: Add AZURE to CLOUDWARE

Let Azure use the image directly built by the release engineering team.

Reviewed by: emaste
Approved by: cperciva (re)
MFC after: 3 days
Sponsored by: The FreeBSD Fo

release: Add AZURE to CLOUDWARE

Let Azure use the image directly built by the release engineering team.

Reviewed by: emaste
Approved by: cperciva (re)
MFC after: 3 days
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D41906

show more ...

release/Makefile.vm: Add cloudware overrides

Allow the cloudware *_FLAVOURS and *_FSLIST values to be overridden
at the command line, to assist users who want to e.g. build only one
of the many EC2

release/Makefile.vm: Add cloudware overrides

Allow the cloudware *_FLAVOURS and *_FSLIST values to be overridden
at the command line, to assist users who want to e.g. build only one
of the many EC2 AMIs available.

show more ...

release/Makefile.vm: Rework emulator-portinstall

The emulator-portinstall target now unconditionally ensures that qemu
is installed; but is only invoked if needed (aka. when cross building
VM images

release/Makefile.vm: Rework emulator-portinstall

The emulator-portinstall target now unconditionally ensures that qemu
is installed; but is only invoked if needed (aka. when cross building
VM images).

MFC After: 3 days
MFC With: 97bd53ef4d20 ("Fix duplicate rc.conf files")

show more ...

Makefile.vm: Fix duplicate rc.conf files

Two bugs in Makefile.vm resulted in disk images being "built" multiple
times, resulting in lines added to /etc/rc.conf being duplicated:

1. The vm-image tar

Makefile.vm: Fix duplicate rc.conf files

Two bugs in Makefile.vm resulted in disk images being "built" multiple
times, resulting in lines added to /etc/rc.conf being duplicated:

1. The vm-image target reused the same "staging tree" directory for all
of its builds (multiple disk image types and multiple filesystem types).

2. The cw-type-flavour-fs target depends on emulator-portinstall, which
did not have a 'touch ${.TARGET}' and thus re-ran every time -- and
caused the cw-type-flavour-fs target to be re-run. This was triggered
by release builds running `make cloudware-release` (creating the disk
images) followed by `make ec2amis` (which re-created the disk images
prior to uploading them).

MFC After: 1 week
Sponsored by: https://www.patreon.com/cperciva

show more ...

EC2: Add experimental cloud-init images

Known issues:

1. The ec2-user user is created with a homedir of /usr/home/ec2-user
instead of /home/ec2-user; this appears to be a bug in cloud-init's
FreeBS

EC2: Add experimental cloud-init images

Known issues:

1. The ec2-user user is created with a homedir of /usr/home/ec2-user
instead of /home/ec2-user; this appears to be a bug in cloud-init's
FreeBSD support.

2. Cloud-init configures IPv4 networking but not IPv6 networking.

releng/14.0 candidate.

Discussed with: gjb
Reviewed by: imp
MFC after: 5 days
Relnotes: yes
Sponsored by: https://www.patreon.com/cperciva
Differential Revision: https://reviews.freebsd.org/D41793

show more ...

EC2: Flavour existing AMIs as "base"

Using the recently-added "cloudware flavours" mechanism, turn the
existing EC2 AMIs into a new "base" flavour. The only user-visible
change is that AMI names no

EC2: Flavour existing AMIs as "base"

Using the recently-added "cloudware flavours" mechanism, turn the
existing EC2 AMIs into a new "base" flavour. The only user-visible
change is that AMI names now include the word "base".

releng/14.0 candidate.

Discussed with: gjb
Reviewed by: imp
MFC after: 5 days
Sponsored by: https://www.patreon.com/cperciva
Differential Revision: https://reviews.freebsd.org/D41791

show more ...

release/Makefile.vm: Add support for CW flavours

There are two "Vagrant" images right now: "Vagrant Image for VirtualBox"
and "Vagrant Image for VMWare". Rather than listing these separately in
a C

release/Makefile.vm: Add support for CW flavours

There are two "Vagrant" images right now: "Vagrant Image for VirtualBox"
and "Vagrant Image for VMWare". Rather than listing these separately in
a CLOUDWARE list, place "VAGRANT" into a CLOUDWARE_TYPES list and then
use a VAGRANT_FLAVOURS variable to identify the two versions. Add make
logic to allow defaults (in this case, image format and filesystem) to
be specified once for VAGRANT and inherited by both flavours.

This will make future work to add flavoured EC2 images simpler.

releng/14.0 candidate.

Discussed with: gjb
Reviewed by: imp
MFC after: 5 days
Sponsored by: https://www.patreon.com/cperciva
Differential Revision: https://reviews.freebsd.org/D41790

show more ...