1.\" Copyright (c) 2013 Peter Grehan 2.\" All rights reserved. 3.\" 4.\" Redistribution and use in source and binary forms, with or without 5.\" modification, are permitted provided that the following conditions 6.\" are met: 7.\" 1. Redistributions of source code must retain the above copyright 8.\" notice, this list of conditions and the following disclaimer. 9.\" 2. Redistributions in binary form must reproduce the above copyright 10.\" notice, this list of conditions and the following disclaimer in the 11.\" documentation and/or other materials provided with the distribution. 12.\" 13.\" THIS SOFTWARE IS PROVIDED BY THE AUTHORS AND CONTRIBUTORS ``AS IS'' AND 14.\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 15.\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE 16.\" ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHORS OR CONTRIBUTORS BE LIABLE 17.\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 18.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS 19.\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) 20.\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT 21.\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY 22.\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF 23.\" SUCH DAMAGE. 24.\" 25.\" $FreeBSD$ 26.\" 27.Dd December 13, 2019 28.Dt BHYVE 8 29.Os 30.Sh NAME 31.Nm bhyve 32.Nd "run a guest operating system inside a virtual machine" 33.Sh SYNOPSIS 34.Nm 35.Op Fl abehuwxACHPSWY 36.Oo 37.Fl c\~ Ns 38.Oo 39.Op Ar cpus= Ns 40.Ar numcpus Ns 41.Oc Ns 42.Op Ar ,sockets=n Ns 43.Op Ar ,cores=n Ns 44.Op Ar ,threads=n 45.Oc 46.Op Fl g Ar gdbport 47.Op Fl l Ar help|lpcdev Ns Op , Ns Ar conf 48.Op Fl m Ar memsize Ns Op Ar K|k|M|m|G|g|T|t 49.Op Fl p Ar vcpu:hostcpu 50.Op Fl s Ar help|slot,emulation Ns Op , Ns Ar conf 51.Op Fl G Ar port 52.Op Fl U Ar uuid 53.Ar vmname 54.Sh DESCRIPTION 55.Nm 56is a hypervisor that runs guest operating systems inside a 57virtual machine. 58.Pp 59Parameters such as the number of virtual CPUs, amount of guest memory, and 60I/O connectivity can be specified with command-line parameters. 61.Pp 62If not using a boot ROM, the guest operating system must be loaded with 63.Xr bhyveload 8 64or a similar boot loader before running 65.Nm , 66otherwise, it is enough to run 67.Nm 68with a boot ROM of choice. 69.Pp 70.Nm 71runs until the guest operating system reboots or an unhandled hypervisor 72exit is detected. 73.Sh OPTIONS 74.Bl -tag -width 10n 75.It Fl a 76The guest's local APIC is configured in xAPIC mode. 77The xAPIC mode is the default setting so this option is redundant. 78It will be deprecated in a future version. 79.It Fl A 80Generate ACPI tables. 81Required for 82.Fx Ns /amd64 83guests. 84.It Fl b 85Enable a low-level console device supported by 86.Fx 87kernels compiled with 88.Cd "device bvmconsole" . 89This option will be deprecated in a future version. 90.It Fl c Op Ar setting ... 91Number of guest virtual CPUs 92and/or the CPU topology. 93The default value for each of 94.Ar numcpus , 95.Ar sockets , 96.Ar cores , 97and 98.Ar threads 99is 1. 100The current maximum number of guest virtual CPUs is 16. 101If 102.Ar numcpus 103is not specified then it will be calculated from the other arguments. 104The topology must be consistent in that the 105.Ar numcpus 106must equal the product of 107.Ar sockets , 108.Ar cores , 109and 110.Ar threads . 111If a 112.Ar setting 113is specified more than once the last one has precedence. 114.It Fl C 115Include guest memory in core file. 116.It Fl e 117Force 118.Nm 119to exit when a guest issues an access to an I/O port that is not emulated. 120This is intended for debug purposes. 121.It Fl g Ar gdbport 122For 123.Fx 124kernels compiled with 125.Cd "device bvmdebug" , 126allow a remote kernel kgdb to be relayed to the guest kernel gdb stub 127via a local IPv4 address and this port. 128This option will be deprecated in a future version. 129.It Fl G Ar port 130Start a debug server that uses the GDB protocol to export guest state to a 131debugger. 132An IPv4 TCP socket will be bound to the supplied 133.Ar port 134to listen for debugger connections. 135Only a single debugger may be attached to the debug server at a time. 136If 137.Ar port 138begins with 139.Sq w , 140.Nm 141will pause execution at the first instruction waiting for a debugger to attach. 142.It Fl h 143Print help message and exit. 144.It Fl H 145Yield the virtual CPU thread when a HLT instruction is detected. 146If this option is not specified, virtual CPUs will use 100% of a host CPU. 147.It Fl l Op Ar help|lpcdev Ns Op , Ns Ar conf 148Allow devices behind the LPC PCI-ISA bridge to be configured. 149The only supported devices are the TTY-class devices 150.Ar com1 151and 152.Ar com2 153and the boot ROM device 154.Ar bootrom . 155.Pp 156.Ar help 157print a list of supported LPC devices. 158.It Fl m Ar memsize Ns Op Ar K|k|M|m|G|g|T|t 159Guest physical memory size in bytes. 160This must be the same size that was given to 161.Xr bhyveload 8 . 162.Pp 163The size argument may be suffixed with one of K, M, G or T (either upper 164or lower case) to indicate a multiple of kilobytes, megabytes, gigabytes, 165or terabytes. 166If no suffix is given, the value is assumed to be in megabytes. 167.Pp 168.Ar memsize 169defaults to 256M. 170.It Fl p Ar vcpu:hostcpu 171Pin guest's virtual CPU 172.Em vcpu 173to 174.Em hostcpu . 175.It Fl P 176Force the guest virtual CPU to exit when a PAUSE instruction is detected. 177.It Fl s Op Ar help|slot,emulation Ns Op , Ns Ar conf 178Configure a virtual PCI slot and function. 179.Pp 180.Nm 181provides PCI bus emulation and virtual devices that can be attached to 182slots on the bus. 183There are 32 available slots, with the option of providing up to 8 functions 184per slot. 185.Bl -tag -width 10n 186.It Ar help 187print a list of supported PCI devices. 188.It Ar slot 189.Ar pcislot[:function] 190.Ar bus:pcislot:function 191.Pp 192The 193.Ar pcislot 194value is 0 to 31. 195The optional 196.Ar function 197value is 0 to 7. 198The optional 199.Ar bus 200value is 0 to 255. 201If not specified, the 202.Ar function 203value defaults to 0. 204If not specified, the 205.Ar bus 206value defaults to 0. 207.It Ar emulation 208.Bl -tag -width 10n 209.It Li hostbridge | Li amd_hostbridge 210.Pp 211Provide a simple host bridge. 212This is usually configured at slot 0, and is required by most guest 213operating systems. 214The 215.Li amd_hostbridge 216emulation is identical but uses a PCI vendor ID of 217.Li AMD . 218.It Li passthru 219PCI pass-through device. 220.It Li virtio-net 221Virtio network interface. 222.It Li virtio-blk 223Virtio block storage interface. 224.It Li virtio-scsi 225Virtio SCSI interface. 226.It Li virtio-rnd 227Virtio RNG interface. 228.It Li virtio-console 229Virtio console interface, which exposes multiple ports 230to the guest in the form of simple char devices for simple IO 231between the guest and host userspaces. 232.It Li ahci 233AHCI controller attached to arbitrary devices. 234.It Li ahci-cd 235AHCI controller attached to an ATAPI CD/DVD. 236.It Li ahci-hd 237AHCI controller attached to a SATA hard-drive. 238.It Li e1000 239Intel e82545 network interface. 240.It Li uart 241PCI 16550 serial device. 242.It Li lpc 243LPC PCI-ISA bridge with COM1 and COM2 16550 serial ports and a boot ROM. 244The LPC bridge emulation can only be configured on bus 0. 245.It Li fbuf 246Raw framebuffer device attached to VNC server. 247.It Li xhci 248eXtensible Host Controller Interface (xHCI) USB controller. 249.It Li nvme 250NVM Express (NVMe) controller. 251.It Li hda 252High Definition Audio Controller. 253.El 254.It Op Ar conf 255This optional parameter describes the backend for device emulations. 256If 257.Ar conf 258is not specified, the device emulation has no backend and can be 259considered unconnected. 260.Pp 261Network devices: 262.Bl -tag -width 10n 263.It Ar tapN Ns Op , Ns Ar mac=xx:xx:xx:xx:xx:xx 264.It Ar vmnetN Ns Op , Ns Ar mac=xx:xx:xx:xx:xx:xx 265.Pp 266If 267.Ar mac 268is not specified, the MAC address is derived from a fixed OUI and the 269remaining bytes from an MD5 hash of the slot and function numbers and 270the device name. 271.Pp 272The MAC address is an ASCII string in 273.Xr ethers 5 274format. 275.El 276.Pp 277Block storage devices: 278.Bl -tag -width 10n 279.It Pa /filename Ns Oo , Ns Ar block-device-options Oc 280.It Pa /dev/xxx Ns Oo , Ns Ar block-device-options Oc 281.El 282.Pp 283The 284.Ar block-device-options 285are: 286.Bl -tag -width 8n 287.It Li nocache 288Open the file with 289.Dv O_DIRECT . 290.It Li direct 291Open the file using 292.Dv O_SYNC . 293.It Li ro 294Force the file to be opened read-only. 295.It Li sectorsize= Ns Ar logical Ns Oo / Ns Ar physical Oc 296Specify the logical and physical sector sizes of the emulated disk. 297The physical sector size is optional and is equal to the logical sector size 298if not explicitly specified. 299.El 300.Pp 301SCSI devices: 302.Bl -tag -width 10n 303.It Pa /dev/cam/ctl Ns Oo Ar pp . Ns Ar vp Oc Ns Oo , Ns Ar scsi-device-options Oc 304.El 305.Pp 306The 307.Ar scsi-device-options 308are: 309.Bl -tag -width 10n 310.It Li iid= Ns Ar IID 311Initiator ID to use when sending requests to specified CTL port. 312The default value is 0. 313.El 314.Pp 315TTY devices: 316.Bl -tag -width 10n 317.It Li stdio 318Connect the serial port to the standard input and output of 319the 320.Nm 321process. 322.It Pa /dev/xxx 323Use the host TTY device for serial port I/O. 324.El 325.Pp 326Boot ROM device: 327.Bl -tag -width 10n 328.It Pa romfile 329Map 330.Ar romfile 331in the guest address space reserved for boot firmware. 332.El 333.Pp 334Pass-through devices: 335.Bl -tag -width 10n 336.It Ns Ar slot Ns / Ns Ar bus Ns / Ns Ar function 337Connect to a PCI device on the host at the selector described by 338.Ar slot , 339.Ar bus , 340and 341.Ar function 342numbers. 343.El 344.Pp 345Guest memory must be wired using the 346.Fl S 347option when a pass-through device is configured. 348.Pp 349The host device must have been reserved at boot-time using the 350.Va pptdevs 351loader variable as described in 352.Xr vmm 4 . 353.Pp 354Virtio console devices: 355.Bl -tag -width 10n 356.It Li port1= Ns Pa /path/to/port1.sock Ns ,anotherport= Ns Pa ... 357A maximum of 16 ports per device can be created. 358Every port is named and corresponds to a Unix domain socket created by 359.Nm . 360.Nm 361accepts at most one connection per port at a time. 362.Pp 363Limitations: 364.Bl -bullet -offset 2n 365.It 366Due to lack of destructors in 367.Nm , 368sockets on the filesystem must be cleaned up manually after 369.Nm 370exits. 371.It 372There is no way to use the "console port" feature, nor the console port 373resize at present. 374.It 375Emergency write is advertised, but no-op at present. 376.El 377.El 378.Pp 379Framebuffer devices: 380.Bl -tag -width 10n 381.It Xo 382.Oo rfb= Ns Oo Ar IP\&: Oc Ns Ar port Oc Ns Oo ,w= Ns Ar width Oc Ns Oo ,h= Ns 383.Ar height Oc Ns Oo ,vga= Ns Ar vgaconf Oc Ns Oo Ns ,wait Oc Ns Oo ,password= Ns 384.Ar password Oc 385.Xc 386.Bl -tag -width 8n 387.It Ar IPv4:port No or Ar [IPv6%zone]:port 388An 389.Ar IP 390address and a 391.Ar port 392VNC should listen on. 393The default is to listen on localhost IPv4 address and default VNC port 5900. 394An IPv6 address must be enclosed in square brackets and may contain an 395optional zone identifier. 396.It Ar width No and Ar height 397A display resolution, width and height, respectively. 398If not specified, a default resolution of 1024x768 pixels will be used. 399Minimal supported resolution is 640x480 pixels, 400and maximum is 1920x1200 pixels. 401.It Ar vgaconf 402Possible values for this option are 403.Dq io 404(default), 405.Dq on 406, and 407.Dq off . 408PCI graphics cards have a dual personality in that they are 409standard PCI devices with BAR addressing, but may also 410implicitly decode legacy VGA I/O space 411.Pq Ad 0x3c0-3df 412and memory space 413.Pq 64KB at Ad 0xA0000 . 414The default 415.Dq io 416option should be used for guests that attempt to issue BIOS calls which result 417in I/O port queries, and fail to boot if I/O decode is disabled. 418.Pp 419The 420.Dq on 421option should be used along with the CSM BIOS capability in UEFI 422to boot traditional BIOS guests that require the legacy VGA I/O and 423memory regions to be available. 424.Pp 425The 426.Dq off 427option should be used for the UEFI guests that assume that 428VGA adapter is present if they detect the I/O ports. 429An example of such a guest is 430.Ox 431in UEFI mode. 432.Pp 433Please refer to the 434.Nm 435.Fx 436wiki page 437.Pq Lk https://wiki.freebsd.org/bhyve 438for configuration notes of particular guests. 439.It wait 440Instruct 441.Nm 442to only boot upon the initiation of a VNC connection, simplifying the 443installation of operating systems that require immediate keyboard input. 444This can be removed for post-installation use. 445.It password 446This type of authentication is known to be cryptographically weak and is not 447intended for use on untrusted networks. 448Many implementations will want to use stronger security, such as running 449the session over an encrypted channel provided by IPsec or SSH. 450.El 451.El 452.Pp 453xHCI USB devices: 454.Bl -tag -width 10n 455.It Li tablet 456A USB tablet device which provides precise cursor synchronization 457when using VNC. 458.El 459.Pp 460NVMe devices: 461.Bl -tag -width 10n 462.It Li devpath 463Accepted device paths are: 464.Ar /dev/blockdev 465or 466.Ar /path/to/image 467or 468.Ar ram=size_in_MiB . 469.It Li maxq 470Max number of queues. 471.It Li qsz 472Max elements in each queue. 473.It Li ioslots 474Max number of concurrent I/O requests. 475.It Li sectsz 476Sector size (defaults to blockif sector size). 477.It Li ser 478Serial number with maximum 20 characters. 479.El 480.Pp 481HD Audio devices: 482.Bl -tag -width 10n 483.It Li play 484Playback device, typically 485.Ar /dev/dsp0 . 486.It Li rec 487Recording device, typically 488.Ar /dev/dsp0 . 489.El 490.El 491.It Fl S 492Wire guest memory. 493.It Fl u 494RTC keeps UTC time. 495.It Fl U Ar uuid 496Set the universally unique identifier 497.Pq UUID 498in the guest's System Management BIOS System Information structure. 499By default a UUID is generated from the host's hostname and 500.Ar vmname . 501.It Fl w 502Ignore accesses to unimplemented Model Specific Registers (MSRs). 503This is intended for debug purposes. 504.It Fl W 505Force virtio PCI device emulations to use MSI interrupts instead of MSI-X 506interrupts. 507.It Fl x 508The guest's local APIC is configured in x2APIC mode. 509.It Fl Y 510Disable MPtable generation. 511.It Ar vmname 512Alphanumeric name of the guest. 513This should be the same as that created by 514.Xr bhyveload 8 . 515.El 516.Sh DEBUG SERVER 517The current debug server provides limited support for debuggers. 518.Ss Registers 519Each virtual CPU is exposed to the debugger as a thread. 520.Pp 521General purpose registers can be queried for each virtual CPU, but other 522registers such as floating-point and system registers cannot be queried. 523.Ss Memory 524Memory (including memory mapped I/O regions) can be read and written by the debugger. 525Memory operations use virtual addresses that are resolved to physical addresses 526via the current virtual CPU's active address translation. 527.Ss Control 528The running guest can be interrupted by the debugger at any time 529.Pq for example, by pressing Ctrl-C in the debugger . 530.Pp 531Single stepping is only supported on Intel CPUs supporting the MTRAP VM exit. 532.Pp 533Breakpoints are supported on Intel CPUs that support single stepping. 534Note that continuing from a breakpoint while interrupts are enabled in the 535guest may not work as expected due to timer interrupts firing while single 536stepping over the breakpoint. 537.Sh SIGNAL HANDLING 538.Nm 539deals with the following signals: 540.Pp 541.Bl -tag -width indent -compact 542.It SIGTERM 543Trigger ACPI poweroff for a VM 544.El 545.Sh EXIT STATUS 546Exit status indicates how the VM was terminated: 547.Pp 548.Bl -tag -width indent -compact 549.It 0 550rebooted 551.It 1 552powered off 553.It 2 554halted 555.It 3 556triple fault 557.It 4 558exited due to an error 559.El 560.Sh EXAMPLES 561If not using a boot ROM, the guest operating system must have been loaded with 562.Xr bhyveload 8 563or a similar boot loader before 564.Xr bhyve 4 565can be run. 566Otherwise, the boot loader is not needed. 567.Pp 568To run a virtual machine with 1GB of memory, two virtual CPUs, a virtio 569block device backed by the 570.Pa /my/image 571filesystem image, and a serial port for the console: 572.Bd -literal -offset indent 573bhyve -c 2 -s 0,hostbridge -s 1,lpc -s 2,virtio-blk,/my/image \\ 574 -l com1,stdio -A -H -P -m 1G vm1 575.Ed 576.Pp 577Run a 24GB single-CPU virtual machine with three network ports, one of which 578has a MAC address specified: 579.Bd -literal -offset indent 580bhyve -s 0,hostbridge -s 1,lpc -s 2:0,virtio-net,tap0 \\ 581 -s 2:1,virtio-net,tap1 \\ 582 -s 2:2,virtio-net,tap2,mac=00:be:fa:76:45:00 \\ 583 -s 3,virtio-blk,/my/image -l com1,stdio \\ 584 -A -H -P -m 24G bigvm 585.Ed 586.Pp 587Run an 8GB quad-CPU virtual machine with 8 AHCI SATA disks, an AHCI ATAPI 588CD-ROM, a single virtio network port, an AMD hostbridge, and the console 589port connected to an 590.Xr nmdm 4 591null-modem device. 592.Bd -literal -offset indent 593bhyve -c 4 \\ 594 -s 0,amd_hostbridge -s 1,lpc \\ 595 -s 1:0,ahci,hd:/images/disk.1,hd:/images/disk.2,\\ 596hd:/images/disk.3,hd:/images/disk.4,\\ 597hd:/images/disk.5,hd:/images/disk.6,\\ 598hd:/images/disk.7,hd:/images/disk.8,\\ 599cd:/images/install.iso \\ 600 -s 3,virtio-net,tap0 \\ 601 -l com1,/dev/nmdm0A \\ 602 -A -H -P -m 8G 603.Ed 604.Pp 605Run a UEFI virtual machine with a display resolution of 800 by 600 pixels 606that can be accessed via VNC at: 0.0.0.0:5900. 607.Bd -literal -offset indent 608bhyve -c 2 -m 4G -w -H \\ 609 -s 0,hostbridge \\ 610 -s 3,ahci-cd,/path/to/uefi-OS-install.iso \\ 611 -s 4,ahci-hd,disk.img \\ 612 -s 5,virtio-net,tap0 \\ 613 -s 29,fbuf,tcp=0.0.0.0:5900,w=800,h=600,wait \\ 614 -s 30,xhci,tablet \\ 615 -s 31,lpc -l com1,stdio \\ 616 -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \\ 617 uefivm 618.Ed 619.Pp 620Run a UEFI virtual machine with a VNC display that is bound to all IPv6 621addresses on port 5900. 622.Bd -literal -offset indent 623bhyve -c 2 -m 4G -w -H \\ 624 -s 0,hostbridge \\ 625 -s 4,ahci-hd,disk.img \\ 626 -s 5,virtio-net,tap0 \\ 627 -s 29,fbuf,tcp=[::]:5900,w=800,h=600 \\ 628 -s 30,xhci,tablet \\ 629 -s 31,lpc -l com1,stdio \\ 630 -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \\ 631 uefivm 632.Ed 633.Sh SEE ALSO 634.Xr bhyve 4 , 635.Xr nmdm 4 , 636.Xr vmm 4 , 637.Xr ethers 5 , 638.Xr bhyvectl 8 , 639.Xr bhyveload 8 640.Pp 641.Rs 642.%A Intel 643.%B 64 and IA-32 Architectures Software Developer’s Manual 644.%V Volume 3 645.Re 646.Sh HISTORY 647.Nm 648first appeared in 649.Fx 10.0 . 650.Sh AUTHORS 651.An Neel Natu Aq Mt neel@freebsd.org 652.An Peter Grehan Aq Mt grehan@freebsd.org 653