xref: /freebsd/usr.sbin/bhyve/bhyve.8 (revision 7029da5c36f2d3cf6bb6c81bf551229f416399e8)
1.\" Copyright (c) 2013 Peter Grehan
2.\" All rights reserved.
3.\"
4.\" Redistribution and use in source and binary forms, with or without
5.\" modification, are permitted provided that the following conditions
6.\" are met:
7.\" 1. Redistributions of source code must retain the above copyright
8.\"    notice, this list of conditions and the following disclaimer.
9.\" 2. Redistributions in binary form must reproduce the above copyright
10.\"    notice, this list of conditions and the following disclaimer in the
11.\"    documentation and/or other materials provided with the distribution.
12.\"
13.\" THIS SOFTWARE IS PROVIDED BY THE AUTHORS AND CONTRIBUTORS ``AS IS'' AND
14.\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
15.\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
16.\" ARE DISCLAIMED.  IN NO EVENT SHALL THE AUTHORS OR CONTRIBUTORS BE LIABLE
17.\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
18.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
19.\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
20.\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
21.\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
22.\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
23.\" SUCH DAMAGE.
24.\"
25.\" $FreeBSD$
26.\"
27.Dd December 13, 2019
28.Dt BHYVE 8
29.Os
30.Sh NAME
31.Nm bhyve
32.Nd "run a guest operating system inside a virtual machine"
33.Sh SYNOPSIS
34.Nm
35.Op Fl abehuwxACHPSWY
36.Oo
37.Fl c\~ Ns
38.Oo
39.Op Ar cpus= Ns
40.Ar numcpus Ns
41.Oc Ns
42.Op Ar ,sockets=n Ns
43.Op Ar ,cores=n Ns
44.Op Ar ,threads=n
45.Oc
46.Op Fl g Ar gdbport
47.Op Fl l Ar help|lpcdev Ns Op , Ns Ar conf
48.Op Fl m Ar memsize Ns Op Ar K|k|M|m|G|g|T|t
49.Op Fl p Ar vcpu:hostcpu
50.Op Fl s Ar help|slot,emulation Ns Op , Ns Ar conf
51.Op Fl G Ar port
52.Op Fl U Ar uuid
53.Ar vmname
54.Sh DESCRIPTION
55.Nm
56is a hypervisor that runs guest operating systems inside a
57virtual machine.
58.Pp
59Parameters such as the number of virtual CPUs, amount of guest memory, and
60I/O connectivity can be specified with command-line parameters.
61.Pp
62If not using a boot ROM, the guest operating system must be loaded with
63.Xr bhyveload 8
64or a similar boot loader before running
65.Nm ,
66otherwise, it is enough to run
67.Nm
68with a boot ROM of choice.
69.Pp
70.Nm
71runs until the guest operating system reboots or an unhandled hypervisor
72exit is detected.
73.Sh OPTIONS
74.Bl -tag -width 10n
75.It Fl a
76The guest's local APIC is configured in xAPIC mode.
77The xAPIC mode is the default setting so this option is redundant.
78It will be deprecated in a future version.
79.It Fl A
80Generate ACPI tables.
81Required for
82.Fx Ns /amd64
83guests.
84.It Fl b
85Enable a low-level console device supported by
86.Fx
87kernels compiled with
88.Cd "device bvmconsole" .
89This option will be deprecated in a future version.
90.It Fl c Op Ar setting ...
91Number of guest virtual CPUs
92and/or the CPU topology.
93The default value for each of
94.Ar numcpus ,
95.Ar sockets ,
96.Ar cores ,
97and
98.Ar threads
99is 1.
100The current maximum number of guest virtual CPUs is 16.
101If
102.Ar numcpus
103is not specified then it will be calculated from the other arguments.
104The topology must be consistent in that the
105.Ar numcpus
106must equal the product of
107.Ar sockets ,
108.Ar cores ,
109and
110.Ar threads .
111If a
112.Ar setting
113is specified more than once the last one has precedence.
114.It Fl C
115Include guest memory in core file.
116.It Fl e
117Force
118.Nm
119to exit when a guest issues an access to an I/O port that is not emulated.
120This is intended for debug purposes.
121.It Fl g Ar gdbport
122For
123.Fx
124kernels compiled with
125.Cd "device bvmdebug" ,
126allow a remote kernel kgdb to be relayed to the guest kernel gdb stub
127via a local IPv4 address and this port.
128This option will be deprecated in a future version.
129.It Fl G Ar port
130Start a debug server that uses the GDB protocol to export guest state to a
131debugger.
132An IPv4 TCP socket will be bound to the supplied
133.Ar port
134to listen for debugger connections.
135Only a single debugger may be attached to the debug server at a time.
136If
137.Ar port
138begins with
139.Sq w ,
140.Nm
141will pause execution at the first instruction waiting for a debugger to attach.
142.It Fl h
143Print help message and exit.
144.It Fl H
145Yield the virtual CPU thread when a HLT instruction is detected.
146If this option is not specified, virtual CPUs will use 100% of a host CPU.
147.It Fl l Op Ar help|lpcdev Ns Op , Ns Ar conf
148Allow devices behind the LPC PCI-ISA bridge to be configured.
149The only supported devices are the TTY-class devices
150.Ar com1
151and
152.Ar com2
153and the boot ROM device
154.Ar bootrom .
155.Pp
156.Ar help
157print a list of supported LPC devices.
158.It Fl m Ar memsize Ns Op Ar K|k|M|m|G|g|T|t
159Guest physical memory size in bytes.
160This must be the same size that was given to
161.Xr bhyveload 8 .
162.Pp
163The size argument may be suffixed with one of K, M, G or T (either upper
164or lower case) to indicate a multiple of kilobytes, megabytes, gigabytes,
165or terabytes.
166If no suffix is given, the value is assumed to be in megabytes.
167.Pp
168.Ar memsize
169defaults to 256M.
170.It Fl p Ar vcpu:hostcpu
171Pin guest's virtual CPU
172.Em vcpu
173to
174.Em hostcpu .
175.It Fl P
176Force the guest virtual CPU to exit when a PAUSE instruction is detected.
177.It Fl s Op Ar help|slot,emulation Ns Op , Ns Ar conf
178Configure a virtual PCI slot and function.
179.Pp
180.Nm
181provides PCI bus emulation and virtual devices that can be attached to
182slots on the bus.
183There are 32 available slots, with the option of providing up to 8 functions
184per slot.
185.Bl -tag -width 10n
186.It Ar help
187print a list of supported PCI devices.
188.It Ar slot
189.Ar pcislot[:function]
190.Ar bus:pcislot:function
191.Pp
192The
193.Ar pcislot
194value is 0 to 31.
195The optional
196.Ar function
197value is 0 to 7.
198The optional
199.Ar bus
200value is 0 to 255.
201If not specified, the
202.Ar function
203value defaults to 0.
204If not specified, the
205.Ar bus
206value defaults to 0.
207.It Ar emulation
208.Bl -tag -width 10n
209.It Li hostbridge | Li amd_hostbridge
210.Pp
211Provide a simple host bridge.
212This is usually configured at slot 0, and is required by most guest
213operating systems.
214The
215.Li amd_hostbridge
216emulation is identical but uses a PCI vendor ID of
217.Li AMD .
218.It Li passthru
219PCI pass-through device.
220.It Li virtio-net
221Virtio network interface.
222.It Li virtio-blk
223Virtio block storage interface.
224.It Li virtio-scsi
225Virtio SCSI interface.
226.It Li virtio-rnd
227Virtio RNG interface.
228.It Li virtio-console
229Virtio console interface, which exposes multiple ports
230to the guest in the form of simple char devices for simple IO
231between the guest and host userspaces.
232.It Li ahci
233AHCI controller attached to arbitrary devices.
234.It Li ahci-cd
235AHCI controller attached to an ATAPI CD/DVD.
236.It Li ahci-hd
237AHCI controller attached to a SATA hard-drive.
238.It Li e1000
239Intel e82545 network interface.
240.It Li uart
241PCI 16550 serial device.
242.It Li lpc
243LPC PCI-ISA bridge with COM1 and COM2 16550 serial ports and a boot ROM.
244The LPC bridge emulation can only be configured on bus 0.
245.It Li fbuf
246Raw framebuffer device attached to VNC server.
247.It Li xhci
248eXtensible Host Controller Interface (xHCI) USB controller.
249.It Li nvme
250NVM Express (NVMe) controller.
251.It Li hda
252High Definition Audio Controller.
253.El
254.It Op Ar conf
255This optional parameter describes the backend for device emulations.
256If
257.Ar conf
258is not specified, the device emulation has no backend and can be
259considered unconnected.
260.Pp
261Network devices:
262.Bl -tag -width 10n
263.It Ar tapN Ns Op , Ns Ar mac=xx:xx:xx:xx:xx:xx
264.It Ar vmnetN Ns Op , Ns Ar mac=xx:xx:xx:xx:xx:xx
265.Pp
266If
267.Ar mac
268is not specified, the MAC address is derived from a fixed OUI and the
269remaining bytes from an MD5 hash of the slot and function numbers and
270the device name.
271.Pp
272The MAC address is an ASCII string in
273.Xr ethers 5
274format.
275.El
276.Pp
277Block storage devices:
278.Bl -tag -width 10n
279.It Pa /filename Ns Oo , Ns Ar block-device-options Oc
280.It Pa /dev/xxx Ns Oo , Ns Ar block-device-options Oc
281.El
282.Pp
283The
284.Ar block-device-options
285are:
286.Bl -tag -width 8n
287.It Li nocache
288Open the file with
289.Dv O_DIRECT .
290.It Li direct
291Open the file using
292.Dv O_SYNC .
293.It Li ro
294Force the file to be opened read-only.
295.It Li sectorsize= Ns Ar logical Ns Oo / Ns Ar physical Oc
296Specify the logical and physical sector sizes of the emulated disk.
297The physical sector size is optional and is equal to the logical sector size
298if not explicitly specified.
299.El
300.Pp
301SCSI devices:
302.Bl -tag -width 10n
303.It Pa /dev/cam/ctl Ns Oo Ar pp . Ns Ar vp Oc Ns Oo , Ns Ar scsi-device-options Oc
304.El
305.Pp
306The
307.Ar scsi-device-options
308are:
309.Bl -tag -width 10n
310.It Li iid= Ns Ar IID
311Initiator ID to use when sending requests to specified CTL port.
312The default value is 0.
313.El
314.Pp
315TTY devices:
316.Bl -tag -width 10n
317.It Li stdio
318Connect the serial port to the standard input and output of
319the
320.Nm
321process.
322.It Pa /dev/xxx
323Use the host TTY device for serial port I/O.
324.El
325.Pp
326Boot ROM device:
327.Bl -tag -width 10n
328.It Pa romfile
329Map
330.Ar romfile
331in the guest address space reserved for boot firmware.
332.El
333.Pp
334Pass-through devices:
335.Bl -tag -width 10n
336.It Ns Ar slot Ns / Ns Ar bus Ns / Ns Ar function
337Connect to a PCI device on the host at the selector described by
338.Ar slot ,
339.Ar bus ,
340and
341.Ar function
342numbers.
343.El
344.Pp
345Guest memory must be wired using the
346.Fl S
347option when a pass-through device is configured.
348.Pp
349The host device must have been reserved at boot-time using the
350.Va pptdevs
351loader variable as described in
352.Xr vmm 4 .
353.Pp
354Virtio console devices:
355.Bl -tag -width 10n
356.It Li port1= Ns Pa /path/to/port1.sock Ns ,anotherport= Ns Pa ...
357A maximum of 16 ports per device can be created.
358Every port is named and corresponds to a Unix domain socket created by
359.Nm .
360.Nm
361accepts at most one connection per port at a time.
362.Pp
363Limitations:
364.Bl -bullet -offset 2n
365.It
366Due to lack of destructors in
367.Nm ,
368sockets on the filesystem must be cleaned up manually after
369.Nm
370exits.
371.It
372There is no way to use the "console port" feature, nor the console port
373resize at present.
374.It
375Emergency write is advertised, but no-op at present.
376.El
377.El
378.Pp
379Framebuffer devices:
380.Bl -tag -width 10n
381.It Xo
382.Oo rfb= Ns Oo Ar IP\&: Oc Ns Ar port Oc Ns Oo ,w= Ns Ar width Oc Ns Oo ,h= Ns
383.Ar height Oc Ns Oo ,vga= Ns Ar vgaconf Oc Ns Oo Ns ,wait Oc Ns Oo ,password= Ns
384.Ar password Oc
385.Xc
386.Bl -tag -width 8n
387.It Ar IPv4:port No or Ar [IPv6%zone]:port
388An
389.Ar IP
390address and a
391.Ar port
392VNC should listen on.
393The default is to listen on localhost IPv4 address and default VNC port 5900.
394An IPv6 address must be enclosed in square brackets and may contain an
395optional zone identifier.
396.It Ar width No and Ar height
397A display resolution, width and height, respectively.
398If not specified, a default resolution of 1024x768 pixels will be used.
399Minimal supported resolution is 640x480 pixels,
400and maximum is 1920x1200 pixels.
401.It Ar vgaconf
402Possible values for this option are
403.Dq io
404(default),
405.Dq on
406, and
407.Dq off .
408PCI graphics cards have a dual personality in that they are
409standard PCI devices with BAR addressing, but may also
410implicitly decode legacy VGA I/O space
411.Pq Ad 0x3c0-3df
412and memory space
413.Pq 64KB at Ad 0xA0000 .
414The default
415.Dq io
416option should be used for guests that attempt to issue BIOS calls which result
417in I/O port queries, and fail to boot if I/O decode is disabled.
418.Pp
419The
420.Dq on
421option should be used along with the CSM BIOS capability in UEFI
422to boot traditional BIOS guests that require the legacy VGA I/O and
423memory regions to be available.
424.Pp
425The
426.Dq off
427option should be used for the UEFI guests that assume that
428VGA adapter is present if they detect the I/O ports.
429An example of such a guest is
430.Ox
431in UEFI mode.
432.Pp
433Please refer to the
434.Nm
435.Fx
436wiki page
437.Pq Lk https://wiki.freebsd.org/bhyve
438for configuration notes of particular guests.
439.It wait
440Instruct
441.Nm
442to only boot upon the initiation of a VNC connection, simplifying the
443installation of operating systems that require immediate keyboard input.
444This can be removed for post-installation use.
445.It password
446This type of authentication is known to be cryptographically weak and is not
447intended for use on untrusted networks.
448Many implementations will want to use stronger security, such as running
449the session over an encrypted channel provided by IPsec or SSH.
450.El
451.El
452.Pp
453xHCI USB devices:
454.Bl -tag -width 10n
455.It Li tablet
456A USB tablet device which provides precise cursor synchronization
457when using VNC.
458.El
459.Pp
460NVMe devices:
461.Bl -tag -width 10n
462.It Li devpath
463Accepted device paths are:
464.Ar /dev/blockdev
465or
466.Ar /path/to/image
467or
468.Ar ram=size_in_MiB .
469.It Li maxq
470Max number of queues.
471.It Li qsz
472Max elements in each queue.
473.It Li ioslots
474Max number of concurrent I/O requests.
475.It Li sectsz
476Sector size (defaults to blockif sector size).
477.It Li ser
478Serial number with maximum 20 characters.
479.El
480.Pp
481HD Audio devices:
482.Bl -tag -width 10n
483.It Li play
484Playback device, typically
485.Ar /dev/dsp0 .
486.It Li rec
487Recording device, typically
488.Ar /dev/dsp0 .
489.El
490.El
491.It Fl S
492Wire guest memory.
493.It Fl u
494RTC keeps UTC time.
495.It Fl U Ar uuid
496Set the universally unique identifier
497.Pq UUID
498in the guest's System Management BIOS System Information structure.
499By default a UUID is generated from the host's hostname and
500.Ar vmname .
501.It Fl w
502Ignore accesses to unimplemented Model Specific Registers (MSRs).
503This is intended for debug purposes.
504.It Fl W
505Force virtio PCI device emulations to use MSI interrupts instead of MSI-X
506interrupts.
507.It Fl x
508The guest's local APIC is configured in x2APIC mode.
509.It Fl Y
510Disable MPtable generation.
511.It Ar vmname
512Alphanumeric name of the guest.
513This should be the same as that created by
514.Xr bhyveload 8 .
515.El
516.Sh DEBUG SERVER
517The current debug server provides limited support for debuggers.
518.Ss Registers
519Each virtual CPU is exposed to the debugger as a thread.
520.Pp
521General purpose registers can be queried for each virtual CPU, but other
522registers such as floating-point and system registers cannot be queried.
523.Ss Memory
524Memory (including memory mapped I/O regions) can be read and written by the debugger.
525Memory operations use virtual addresses that are resolved to physical addresses
526via the current virtual CPU's active address translation.
527.Ss Control
528The running guest can be interrupted by the debugger at any time
529.Pq for example, by pressing Ctrl-C in the debugger .
530.Pp
531Single stepping is only supported on Intel CPUs supporting the MTRAP VM exit.
532.Pp
533Breakpoints are supported on Intel CPUs that support single stepping.
534Note that continuing from a breakpoint while interrupts are enabled in the
535guest may not work as expected due to timer interrupts firing while single
536stepping over the breakpoint.
537.Sh SIGNAL HANDLING
538.Nm
539deals with the following signals:
540.Pp
541.Bl -tag -width indent -compact
542.It SIGTERM
543Trigger ACPI poweroff for a VM
544.El
545.Sh EXIT STATUS
546Exit status indicates how the VM was terminated:
547.Pp
548.Bl -tag -width indent -compact
549.It 0
550rebooted
551.It 1
552powered off
553.It 2
554halted
555.It 3
556triple fault
557.It 4
558exited due to an error
559.El
560.Sh EXAMPLES
561If not using a boot ROM, the guest operating system must have been loaded with
562.Xr bhyveload 8
563or a similar boot loader before
564.Xr bhyve 4
565can be run.
566Otherwise, the boot loader is not needed.
567.Pp
568To run a virtual machine with 1GB of memory, two virtual CPUs, a virtio
569block device backed by the
570.Pa /my/image
571filesystem image, and a serial port for the console:
572.Bd -literal -offset indent
573bhyve -c 2 -s 0,hostbridge -s 1,lpc -s 2,virtio-blk,/my/image \\
574  -l com1,stdio -A -H -P -m 1G vm1
575.Ed
576.Pp
577Run a 24GB single-CPU virtual machine with three network ports, one of which
578has a MAC address specified:
579.Bd -literal -offset indent
580bhyve -s 0,hostbridge -s 1,lpc -s 2:0,virtio-net,tap0 \\
581  -s 2:1,virtio-net,tap1 \\
582  -s 2:2,virtio-net,tap2,mac=00:be:fa:76:45:00 \\
583  -s 3,virtio-blk,/my/image -l com1,stdio \\
584  -A -H -P -m 24G bigvm
585.Ed
586.Pp
587Run an 8GB quad-CPU virtual machine with 8 AHCI SATA disks, an AHCI ATAPI
588CD-ROM, a single virtio network port, an AMD hostbridge, and the console
589port connected to an
590.Xr nmdm 4
591null-modem device.
592.Bd -literal -offset indent
593bhyve -c 4 \\
594  -s 0,amd_hostbridge -s 1,lpc \\
595  -s 1:0,ahci,hd:/images/disk.1,hd:/images/disk.2,\\
596hd:/images/disk.3,hd:/images/disk.4,\\
597hd:/images/disk.5,hd:/images/disk.6,\\
598hd:/images/disk.7,hd:/images/disk.8,\\
599cd:/images/install.iso \\
600  -s 3,virtio-net,tap0 \\
601  -l com1,/dev/nmdm0A \\
602  -A -H -P -m 8G
603.Ed
604.Pp
605Run a UEFI virtual machine with a display resolution of 800 by 600 pixels
606that can be accessed via VNC at: 0.0.0.0:5900.
607.Bd -literal -offset indent
608bhyve -c 2 -m 4G -w -H \\
609  -s 0,hostbridge \\
610  -s 3,ahci-cd,/path/to/uefi-OS-install.iso \\
611  -s 4,ahci-hd,disk.img \\
612  -s 5,virtio-net,tap0 \\
613  -s 29,fbuf,tcp=0.0.0.0:5900,w=800,h=600,wait \\
614  -s 30,xhci,tablet \\
615  -s 31,lpc -l com1,stdio \\
616  -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \\
617   uefivm
618.Ed
619.Pp
620Run a UEFI virtual machine with a VNC display that is bound to all IPv6
621addresses on port 5900.
622.Bd -literal -offset indent
623bhyve -c 2 -m 4G -w -H \\
624  -s 0,hostbridge \\
625  -s 4,ahci-hd,disk.img \\
626  -s 5,virtio-net,tap0 \\
627  -s 29,fbuf,tcp=[::]:5900,w=800,h=600 \\
628  -s 30,xhci,tablet \\
629  -s 31,lpc -l com1,stdio \\
630  -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \\
631   uefivm
632.Ed
633.Sh SEE ALSO
634.Xr bhyve 4 ,
635.Xr nmdm 4 ,
636.Xr vmm 4 ,
637.Xr ethers 5 ,
638.Xr bhyvectl 8 ,
639.Xr bhyveload 8
640.Pp
641.Rs
642.%A Intel
643.%B 64 and IA-32 Architectures Software Developer’s Manual
644.%V Volume 3
645.Re
646.Sh HISTORY
647.Nm
648first appeared in
649.Fx 10.0 .
650.Sh AUTHORS
651.An Neel Natu Aq Mt neel@freebsd.org
652.An Peter Grehan Aq Mt grehan@freebsd.org
653