.\"
.\" This file and its contents are supplied under the terms of the
.\" Common Development and Distribution License ("CDDL"), version 1.0.
.\" You may only use this file in accordance with the terms of version
.\" 1.0 of the CDDL.
.\"
.\" A full copy of the text of the CDDL should have accompanied this
.\" source.  A copy of the CDDL is also available via the Internet at
.\" http://www.illumos.org/license/CDDL.
.\"
.\"
.\" Copyright 2015 Joyent, Inc.
.\"
.Dd March 30, 2022
.Dt OVERLAY 7
.Os
.Sh NAME
.Nm overlay
.Nd Overlay Devices
.Sh DESCRIPTION
Overlay devices are a GLDv3 device that allows users to create overlay
networks that can be used to form the basis of network virtualization
and software defined networking.
Overlay networks allow a single physical network, often called an
.Sy underlay
network, to provide the means for creating multiple logical, isolated,
and discrete layer two and layer three networks on top of it.
.Pp
Overlay devices are administered through
.Xr dladm 8 .
Overlay devices themselves cannot be plumbed up with
.Sy IP ,
.Sy vnd ,
or any other protocol.
Instead, like an
.Sy etherstub ,
they allow for VNICs to be created on top of them.
Like an
.Sy etherstub ,
an overlay device acts as a local switch; however, when it encounters a
non-local destination address, it instead looks up where it should send
the packet, encapsulates it, and sends it out another interface in the
system.
.Pp
A single overlay device encapsulates the logic to answer two different,
but related, questions:
.Pp
.Bl -enum -offset indent -compact
.It
How should a packet be transformed and put on the wire?
.It
Where should a transformed packet be sent?
.El
.Pp
Each of these questions is answered by a plugin.
The first question is answered by what's called an
.Em encapsulation plugin .
The second question is answered by what's called a
.Em search plugin .
Packets are encapsulated and decapsulated using the encapsulation plugin
by the kernel.
The search plugins are all user land plugins that are consumed by the
varpd service whose FMRI is
.Em svc:/network/varpd:default .
This separation allows for the kernel to be responsible for the data
path, while having the search plugins in userland allows the system to
provide a much more expressive interface.
.Ss Overlay Types
Overlay devices come in two different flavors, one where all packets are
always sent to a single address, the other, where the destination of a
packet varies based on the target MAC address of the packet.
This information is maintained in a
.Em target table ,
which is independent and unique to each overlay device.
We call the plugins that send traffic to a single location, for example
a single unicast or multicast IP address, a
.Sy point to point
overlay and the overlay devices that can send traffic to different
locations based on the MAC address of that packet a
.Sy dynamic
overlay.
The plugin type is determined based on the type of the
.Sy search plugin .
These are all fully listed in the section
.Sx Plugins and their Properties .
.Ss Overlay Destination
Both encapsulation and search plugins define the kinds of destinations
that they know how to support.
An encapsulation plugin always has a single destination type that's
determined based on how the encapsulation is defined.
A search plugin, on the other hand, can support multiple combinations of
destinations.
A search plugin must support the destination type of the encapsulation
device.
The destination may require any of the following three pieces of
information, depending on the encapsulation plugin:
.Bl -hang -width Ds
.It Sy MAC Address
.Bd -filled -compact
An Ethernet MAC address is required to determine the destination.
.Ed
.It Sy IP Address
.Bd -filled -compact
An IP address is required.
Both IPv4 and IPv6 addresses are supported.
.Ed
.It Sy Port
.Bd -filled -compact
An IP protocol level (TCP, UDP, SCTP, etc.) port is required.
.Ed
.El
.Pp
The list of destination types that are supported by both the search and
encapsulation plugins is listed in the section
.Sx Plugins and their Properties .
.Ss varpd
The varpd service, mentioned above, is responsible for providing the
virtual ARP daemon.
Its responsibility is conceptually similar to ARP.
It runs all instances of search plugins in the system and is responsible
for answering the kernel's ARP-like questions for where packets should
be sent.
.Pp
The varpd service, svc:/network/varpd:default, must be enabled for
overlay devices to function.
If it is disabled while there are active devices, then most overlay
devices will not function correctly and likely will end up dropping
traffic.
.Sh PLUGINS AND PROPERTIES
Properties fall into three categories in the system:
.Bl -enum -offset indent -compact
.It
Generic properties all overlay devices have
.It
Properties specific to the encapsulation plugin
.It
Properties specific to the search plugin
.El
.Pp
Each property in the system has the following attributes, which mirror
the traditional
.Xr dladm 8
link properties:
.Bl -hang -width Ds
.It Sy Name
.Bd -filled -compact
The name of a property is namespaced by its module and always structured
and referred to as as module/property.
This allows for both an encapsulation and search plugin to have a
property with the same name.
Properties that are valid for all overlay devices and not specific to a
module do not generally use a module prefix.
.Pp
For example, the property
.Sy vxlan/listen_ip
is associated with the
.Sy vxlan
encapsulation module.
.Ed
.It Sy Type
.Bd -filled -compact
Each property in the system has a type.
.Xr dladm 8
takes care of converting between the internal representation and a
value, but the type influences the acceptable input range.
The types are:
.Bl -hang -width Ds
.It Sy INT
A signed integer that is up to eight bytes long
.Pq Sy int64_t .
.It Sy UINT
An unsigned integer that is up to eight bytes long
.Pq Sy uint64_t .
.It Sy IP
Either an IPv4 or IPv6 address in traditional string form.
For example, 192.168.128.23 or 2001:470:8af4::1:1.
IPv4 addresses may also be encoded as IPv4-mapped IPv6 addresses.
.It Sy STRING
A string of ASCII or UTF-8 encoded characters terminated with a
.Sy NUL
byte.
The maximum string length, including the terminator, is currently
256 bytes.
.El
.Ed
.It Sy Permissions
.Bd -filled -compact
Each property has permissions associated with it, which indicate whether
the system considers them read-only properties or read-write properties.
A read-only property can never be updated once the device is created.
This generally includes things like the overlay's encapsulation module.
.Ed
.It Sy Required
.Bd -filled -compact
This property indicates whether the property is required for the given
plugin.
If it is not specified during a call to
.Sy dladm create-overlay ,
then the overlay cannot be successfully created.
Properties which have a
.Sy default
will use that value if one is not specified rather than cause the
overlay creation to fail.
.Ed
.It Sy Current Value
.Bd -filled -compact
The current value of a property, if the property has a value set.
Required properties always have a value set.
.Ed
.It Sy Default Value
.Bd -filled -compact
The default value is an optional part of a given property.
If a property does define a default value, then it will be used when an
overlay is created and no other value is given.
.Ed
.It Sy Value ranges
.Bd -filled -compact
Value ranges are an optional part of a given property.
They indicate a range or set of values that are valid and may be set for
a property.
A property may not declare such a range as it may be impractical or
unknown.
For example, most properties based on IP addresses will not
declare a range.
.Ed
.El
.Pp
The following sections describe both the modules and the properties that
exist for each module, noting their name, type, permissions, whether or
not they are required, and if there is a default value.
In addition, the effects of each property will be described.
.Ss Encapsulation Plugins
.Bl -hang -width Ds
.It Sy vxlan
The
.Sy vxlan
module is a UDP based encapsulation method.
It takes a frame that would be put on the wire, wraps it up in a VXLAN
header and places it in a UDP packet that gets sent out on the
underlying network.
For more details about the specific format of the VXLAN header, see
.Xr vxlan 4P .
.Pp
The
.Sy vxlan
module requires both an
.Sy IP address
and
.Sy port
to address it.
It has a 24-bit virtual network ID space, allowing for
virtual network identifiers that range from
.Sy 0
-
.Sy 16777215 .
.Pp
The
.Sy vxlan
module has the following properties:
.Bl -hang -width Ds
.It Sy vxlan/listen_ip
.Bd -filled -compact
Type:
.Sy IP |
Permissions:
.Sy Read/Write |
.Sy Required
.Ed
.Bd -filled
The
.Sy vxlan/listen_ip
property determines the IP address that the system will accept VXLAN
encapsulated packets on for this overlay.
.Ed
.It Sy vxlan/listen_port
.Bd -filled -compact
Type:
.Sy UINT |
Permissions:
.Sy Read/Write |
.Sy Required
.Ed
.Bd -filled -compact
Default Value:
.Sy 4789 |
Range:
.Sy 0 - 65535
.Ed
.Bd -filled
The
.Sy vxlan/listen_port
property determines the UDP port that the system will listen on for
VXLAN traffic for this overlay.
The default value is
.Sy 4789 ,
the IANA assigned port for VXLAN.
.Ed
.El
.Pp
The
.Sy vxlan/listen_ip
and
.Sy vxlan/listen_port
properties determine how the system will accept VXLAN encapsulated
packets for this interface.
It does not determine the interface that packets will be sent out over.
Multiple overlays that all use VXLAN can share the same IP and port
combination, as the virtual network identifier can be used to tell the
different overlays apart.
.El
.Ss Search Plugins
Because search plugins may support multiple destinations, they may have
more properties listed than necessarily show up for a given overlay.
For example, the
.Sy direct
plugin supports destinations that are identified by both an IP address
and a port, or just an IP address.
In cases where the device is created over an overlay that only uses an
IP address for its destination, then it will not have the
.Sy direct/dest_port
property.
.Bl -hang -width Ds
.It Sy direct
The
.Sy direct
plugin is a point to point module that can be used to create an overlay
that forwards all non-local traffic to a single destination.
It supports destinations that are a combination of an
.Sy IP Address
and a
.Sy port .
.Pp
The
.Sy direct
plugin has the following properties:
.Bl -hang -width Ds
.It Sy direct/dest_ip
.Bd -filled -compact
Type:
.Sy IP |
Permissions:
.Sy Read/Write |
.Sy Required
.Ed
.Bd -filled
The
.Sy direct/dest_ip
property indicates the IP address that all traffic will be sent out.
Traffic will be sent out the corresponding interface based on
traditional IP routing rules and the configuration of the networking
stack of the global zone.
.Ed
.It Sy direct/dest_port
.Bd -filled -compact
Type:
.Sy UINT |
Permissions:
.Sy Read/Write |
.Sy Required
.Ed
.Bd -filled -compact
Default Value:
.Sy - |
Range:
.Sy 0 - 65535
.Ed
.Bd -filled
The
.Sy direct/dest_port
property indicates the TCP or UDP port that all traffic will be directed
to.
.Ed
.El
.It Sy files
The
.Sy files
plugin implements a
.Sy dynamic
plugin that specifies where traffic should be sent based on a file.
It is a glorified version of /etc/ethers.
The
.Sy dynamic
plugin does not support broadcast or multicast traffic, but it has
support for proxy ARP, NDP, and DHCPv4.
For the full details of the file format, see
.Xr overlay_files 5 .
.Pp
The
.Sy files
plugin has the following property:
.Bl -hang -width Ds
.It Sy files/config
.Bd -filled -compact
Type:
.Sy String |
Permissions:
.Sy Read/Write |
.Sy Required
.Ed
.Bd -filled
The
.Sy files/config
property specifies an absolute path to a file to read.
The file is a JSON file that is formatted according to
.Xr overlay_files 5 .
.Ed
.El
.El
.Ss General Properties
Each overlay has the following properties which are used to give
additional information about the system.
None of these properties may be specified as part of a
.Sy dladm create-overlay ,
instead they come from other arguments or from internal parts of the
system.
.Bl -hang -width Ds
.It Sy encap
.Bd -filled -compact
.Sy String |
Permissions:
.Sy Read Only
.Ed
.Bd -filled
The
.Sy encap
property contains the name of the encapsulation module that's in use.
.Ed
.It Sy mtu
.Bd -filled -compact
.Sy UINT |
Permissions:
.Sy Read/Write
.Ed
.Bd -filled -compact
Default Value:
.Sy 1400 |
Range:
.Sy 576 - 9000
.Ed
.Bd -filled
The
.Sy mtu
property describes the maximum transmission unit of the overlay.
The default value is
.Sy 1400
bytes, which ensures that in a traditional deployment with an MTU of
1500 bytes, the overhead that is added from encapsulation is all
accounted for.
It is the administrator's responsibility to ensure that
the device's MTU and the encapsulation overhead does not exceed that of
the interfaces that the encapsulated traffic will be sent out of.
.Pp
To modify the
.Sy mtu
property, use
.Sy dladm set-linkprop .
.Ed
.It Sy search
.Bd -filled -compact
.Sy String |
Permissions:
.Sy Read Only
.Ed
.Bd -filled
The
.Sy search
property contains the name of the search plugin that's in use.
.Ed
.It Sy varpd/id
.Bd -filled -compact
.Sy String |
Permissions:
.Sy Read Only
.Ed
.Bd -filled
The
.Sy varpd/id
property indicates the identifier which the
.Sy varpd
service uses for this overlay.
.Ed
.It Sy vnetid
.Bd -filled -compact
.Sy UINT |
Permissions:
.Sy Read/Write
.Ed
.Bd -filled
The
.Sy vnetid
property has the virtual network identifier that belongs to this overlay.
The valid range for the virtual network identifier depends on the
encapsulation engine.
.Ed
.El
.Sh FMA INTEGRATION
Overlay devices are wired into FMA, the illumos fault management
architecture, and generates error reports depending on the
.Sy search
plugin in use.
Due to limitations in FMA today, when a single overlay
enters a degraded state, meaning that it cannot properly perform look
ups or another error occurred, then it degrades the overall
.Sy overlay
pseudo-device driver.
.Pp
For more fine-grained information about which overlay is actually in a
.Em degraded
state, one should run
.Sy dladm show-overlay -f .
In addition, for each overlay in a degraded state a more useful
diagnostic message is provided which describes the reason that caused
this overlay to enter into a degraded state.
.Pp
The overlay driver is self-healing.
If the problem corrects itself on its own, it will clear the fault on
the corresponding device.
.Sh SEE ALSO
.Xr vxlan 4P ,
.Xr overlay_files 5 ,
.Xr dladm 8