CDDL HEADER START

The contents of this file are subject to the terms of the
Common Development and Distribution License (the "License").
You may not use this file except in compliance with the License.

You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
or http://www.opensolaris.org/os/licensing.
See the License for the specific language governing permissions
and limitations under the License.

When distributing Covered Code, include this CDDL HEADER in each
file and include the License file at usr/src/OPENSOLARIS.LICENSE.
If applicable, add the following below this CDDL HEADER, with the
fields enclosed by brackets "[]" replaced with your own identifying
information: Portions Copyright [yyyy] [name of copyright owner]

CDDL HEADER END

Copyright 2010 Sun Microsystems, Inc.  All rights reserved.
Use is subject to license terms.

Implementation Overview for the NetWork AutoMagic daemon
John Beck, Renee Danson, Michael Hunter, Alan Maguire, Kacheong Poon,
Garima Tripathi, Jan Xie, Anurag Maskey
[Structure and some content shamelessly stolen from Peter Memishian's
dhcpagent architecture overview.]

INTRODUCTION
============

Details about the NWAM requirements, architecture, and design are
available via the NWAM opensolaris project at
http://opensolaris.org/os/project/nwam.  The point of this document is
to place details relevant to somebody attempting to understand the
implementation close to the source code.

THE BASICS
==========

SOURCE FILE ORGANIZATION
=======================
event sources:
	dlpi_events.c
	routing_events.c
	sysevent_events.c

object-specific event handlers:
	enm.c
	known_wlans.c
	loc.c
	ncp.c
	ncu_ip.c
	ncu_phys.c

legacy config upgrade
	llp.c

generic code:
	objects.c
	events.c
	conditions.c
	logging.c
	util.c

nwam door requests:
	door_if.c

entry point:
	main.c

OVERVIEW
========

Here we discuss the essential objects and subtle aspects of the NWAM
daemon implementation.  Note that there is of course much more that is
not discussed here, but after this overview you should be able to fend
for yourself in the source code.

Events and Objects
==================

Events come to NWAM from a variety of different sources asyncronously.

o	routing socket
o	dlpi
o	sysevents
o	doors

Routing sockets and dlpi (DL_NOTE_LINK_UP|DOWN events) are handled by
dedicated threads.  Sysevents and doors are both seen as callbacks into
the process proper and will often post their results to the main event
queue.  All event sources post events onto the main event queue.  In
addition state changes of objects and door requests (requesting current
state or a change of state, specification of a WiFi key etc) can
lead to additional events.  We have daemon-internal events (object
initialization, periodic state checks) which are simply enqueued
on the event queue, and external events which are both enqueued on
the event queue and sent to registered listeners (via nwam_event_send()).

So the structure of the daemon is a set of threads that drive event
generation.  Events are posted either directly onto the event queue
or are delayed by posting onto the pending event queue.  SIGALARMs
are set for the event delay,  and when the SIGALARM is received
pending events that have expired are moved onto the event queue
proper.  Delayed enqueueing is useful for periodic checks.

Decisions to change conditions based upon object state changes are
delayed until after bursts of events.  This is achieved by marking a
flag when it is deemed checking is necessary and then the next time the
queue is empty performing those checks.  A typical event profile will
be one event (e.g. a link down) causing a flurry of other events (e.g.
related interface down).  By waiting until all the consequences of the
initial event have been carried out to make higher level decisions we
implicitly debounce those higher level decisions.

At the moment queue quiet actually means that the queue has been quiet
for some short period of time (.1s).  Typically the flurry of events we
want to work through are internally generated and are back to back in
the queue.  We wait a bit longer in case there are reprucussions from
what we do that cause external events to be posted on us.  We are not
interested in waiting for longer term things to happen but merely to
catch immediate changes.

When running, the daemon will consist of a number of threads:

o	the event handling thread: a thread blocking until events appear on the
	event queue, processing each event in order.  Events that require
	time-consuming processing are spawned in worker threads (e.g. WiFi
	connect, DHCP requests etc).
o	door request threads: the door infrastructure manages server threads
	which process synchronous NWAM client requests (e.g. get state of an
	object, connect to a specific WLAN, initiate a scan on a link etc).
o	various wifi/IP threads: threads which do asynchronous work such as
	DHCP requests, WLAN scans etc that cannot hold up event processing in
	the main event handling thread.
o	routing socket threads: process routing socket messages of interest
	(address additons/deletions) and package them as NWAM messages.
o	dlpi threads: used to monitor for DL_NOTE_LINK messages on links

The daemon is structured around a set of objects representing NCPs[1],
NCUs[2], ENMs[3] and known WLANs and a set of state machines which
consume events which act on those objects.  Object lists are maintained
for each object type, and these contain both a libnwam handle (to allow
reading the object directly) and an optional object data pointer which
can point to state information used to configure the object.

Events can be associated with specific objects (e.g. link up), or associated
with no object in particular (e.g. shutdown).

Each object type registers a set of event handler functions with the event
framework such that when an event occurs, the appropriate handler for the
object type is used.  The event handlers are usually called
nwamd_handle_*_event().

[1] NCP Network Configuration Profile; the set of link- and IP-layer
configuration units which collectively specify how a system should be
connected to the network

[2] NCU Network Configuration Unit; the individual components of an NCP

[3] ENM External Network Modifiers; user executable scripts often used
to configure a VPN

Doors and External Events
=========================

The command interface to nwamd is thread a door at NWAM_DOOR
(/etc/svc/volatile/nwam/nwam_door).  This door allows external program to send
messages to nwamd.  The way doors work is to provide a mechanism for
another process to execute code in your process space.  This looks like
a CSPish send/receive/reply in that the receiving process provide a
syncronization point (via door_create(3C)), the calling process uses
that syncronization point to rendezvous with and provide arguments (via
door_call(3C), and then the receive process reply (via
door_return(3C))) passing back data as required.  The OS makes it such
that the memory used to pass data via door_call(3C) is mapped into the
receiving process which can write back into it and then transparently
have it mapped back to the calling process.

As well as handling internal events of interest, the daemon also needs
to send events of interest (link up/down, WLAN scan/connect results etc)
to (possibly) multiple NWAM client listeners.  This is done via
System V message queues. On registering for events via a libnwam door
request into the daemon (nwam_events_register()), a per-client
(identified by pid) message queue file is created.  The
daemon sends messages to all listeners by examining the list of
message queue files (allowing registration to be robust across
daemon restarts) and sending events to each listener.  This is done
via the libnwam function nwam_event_send() which hides the IPC
mechanism from the daemon.

Objects
=======
Four object lists are maintained within the daemon - one each for
the configuration objects libnwam manages. i.e.:

o	ENMs
o	locations
o	known WLANs
o	NCUs of the current active NCP

Objects have an associated libnwam handle and an optional data
field (which is used for NCUs only).

Locking is straightforward - nwamd_object_init() will initialize
an object of a particular type in the appropriate object list,
returning it with the object lock held. When it is no longer  needed,
nwamd_object_unlock() should be called on the object.

To retrieve an existing object, nwamd_object_find() should be
called - again this returns the object in a locked state.

nwamd_object_lock() is deliberately not exposed outside of objects.c,
since object locking is implicit in the above creation/retrieval
functions.

An object is removed from the object list (with handle destroyed)
via nwamd_object_fini() - the object data (if any) is returned
from this call to allow deallocation.

Object state
============
nwamd deals with 3 broad types of object that need to maintain
internal state: NCUs, ENMs and locations (known WLANs are configuration
objects but don't have a state beyond simply being present).
NWAM objects all share a basic set of states:

State		Description
=====		===========
uninitialized	object representation not present on system or in nwamd
initialized	object representation present in system and in nwamd
disabled	disabled manually
offline		external conditions are not satisfied
offline*	external conditions are satisfied, trying to move online
online*		external conditions no longer satisfied, trying to move offline
online		conditions satisfied and configured
maintenance	error occurred in applying configuration

These deliberately mimic SMF states.

The states of interest are offline, offline* and online.

An object (link/interface NCU, ENM or location) should only move online
when its conditions are satisfied _and_ its configuration has been successfully
applied. This occurs when an ENM method has run or a link is up, or an
interface has at least one address assigned.

To understand the distinction between offline and offline*, consider the case
where a link is of prioritized activation, and either is a lower priority
group - and hence inactive (due to cable being unplugged  or inability to
connect to wifi) - or a higher priority group - and hence active. In general,
we want to distinguish between two cases:

1) when we are actively configuring the link with a view to moving online
(offline*), as would be the case when the link's priority group is
active.
2) when external policy-based conditions prevent a link from being active.
offline should be used for such cases. Links in priority groups above and
below the currently-active group will be offline, since policy precludes them
from activating (as less-prioritized links).

So we see that offline and offline* can thus be used to distinguish between
cases that have the potentiality to move online (offline*) from a policy
perspective - i.e. conditions on the location allow it, or link prioritization
allows it - and cases where external conditions dictate that it should not
(offline).

Once an object reaches offline*, its configuration processes should kick in.
This is where auxiliary state is useful, as it allows us to distinguish between
various states in that configuration process. For example, a link can be
waiting for WLAN selection or key data, or an interface can be waiting for
DHCP response. This auxiliary state can then also be used diagnostically by
libnwam consumers to determine the current status of a link, interface, ENM
etc.

WiFi links present a problem however. On the one hand, we want them
to be inactive when they are not part of the current priority grouping,
while on the other we want to watch out for new WLANs appearing in
scan data if the WiFi link is of a higher priority than the currently-selected
group. The reason we watch out for these is they represent the potential
to change priority grouping to a more preferred group.  To accommodate this,
WiFi links of the same or lower (more preferred) priority group will always
be trying to connect (and thus be offline* if they cannot).

It might appear unnecessary to have a separate state value/machine for
auxiliary state - why can't we simply add the auxiliary state machine to the
global object state machine? Part of the answer is that there are times we
need to run through the same configuration state machine when the global
object state is different - in paticular either offline* or online. Consider
WiFi - we want to do periodic scans to find a "better" WLAN - we can easily
do this by running back through the link state machine of auxiliary
states, but we want to stay online while we do it, since we are still
connected (if the WLAN disconnects of course we go to LINK_DOWN and offline).

Another reason we wish to separate the more general states (offline, online
etc) from the more specific ones (WIFI_NEED_SELECTION etc) is to ensure
that the representation of configuration objects closely matches the way
SMF works.

For an NCU physical link, the following link-specific auxiliary states are
used:

Auxiliary state			Description
===============			===========

LINK_WIFI_SCANNING		Scan in progress
LINK_WIFI_NEED_SELECTION	Need user to specify WLAN
LINK_WIFI_NEED_KEY		Need user to specify a WLAN key for selection
LINK_WIFI_CONNECTING		Connecting to current selection

A WiFI link differs from a wired one in that it always has the
potential to be available - it just depends if visited WLANs are in range.
So such links - if they are higher in the priority grouping than the
currently-active priority group - should always be able to scan, as they
are always "trying" to be activated.

Wired links that do not support  DL_NOTE_LINK_UP/DOWN are problematic,
since we have to simply assume a cable is plugged in.  If an IP NCU
is activated above such a link, and that NCU uses DHCP, a timeout
will be triggered eventually (user-configurable via the nwamd/ncu_wait_time
SMF property of the network/physical:nwam instance) which will cause
us to give up on the link.

For an IP interface NCU, the following auxiliary states are suggested.

Auxiliary state				Description
===============				===========

NWAM_AUX_STATE_IF_WAITING_FOR_ADDR	Waiting for an address to be assigned
NWAM_AUX_STATE_IF_DHCP_TIMED_OUT	DHCP timed out on interface

A link can have multiple logical interfaces plumbed on it consisting
of a mix of static and DHCP-acquired addresses. This means that
we need to decide how to aggregate the state of these logical
interfaces into the NCU state. The concept of "up" we use here
does not correspond to IFF_UP or IFF_RUNNING, but rather
when we get (via getting RTM_NEWADDR events with non-zero
addresses) at least one address assigned to the link.

We use this concept of up as it represents the potential for
network communication - e.g. after assigning a static
address, if the location specifies nameserver etc, it
is possible to communicate over the network. One important
edge case here is that when DHCP information comes
in, we need to reassess location activation conditions and
possibly change or reapply the current location. The problem
is that if we have a static/DHCP mix, and if we rely on
the IP interface's notion of "up" to trigger location activation,
we will likely first apply the location when the static address
has been assigned and before the DHCP information has
been returned (which may include nameserver info). So
the solution is that on getting an RTM_NEWADDR, we
check if the (logical) interface associated is DHCP, and
even if the interface NCU is already up, we reassess
location activation. This will lead to a reapplication of
the current location or possibly a location switch.

In order to move through the various states, a generic
API is supplied

nwam_error_t
nwamd_object_set_state(nwamd_object_t obj, nwamd_state_t state,
    nwamd_aux_state_t aux_state);

This function creates an OBJECT_STATE event containing
the new state/aux_state and enqueues it in the event
queue. Each object registers its own handler for this
event, and in response to the current state/aux state and
desired aux state it responds appropriately in the event
handling thread, spawning other threads to carry out
actions as appropriate. The object state event is
then sent to any registered listeners.

So for NCUs, we define a handle_object_state() function
to run the state machine for the NCU object.

Link state and NCP policy
=========================

NCPs can be either:

o	prioritized: where the constituent link NCUs specify priority group
	numbers (where lower are more favoured) and grouping types.  These
	are used to allow link NCUs to be either grouped separately (exclusive)
	or together (shared or all).
o	manual: their activation is governed by the value of their enabled
	property.
o	a combination of the above.

IP interface NCUs interit their activation from the links below them,
so an IP interface NCU will be active if its underlying link is (assuming
it hasn't been disabled).

At startup, and at regular intervals (often triggered by NWAM
events), the NCP policy needs to be reassessed. There
are a number of causes for NCP policy to be reassessed -

o	a periodic check of link state that occurs every N seconds
o	a link goes from offline(*) to online (cable plug/wifi connect)
o	a link goes from online to offline (cable unplug/wifi disconnect).

Any of these should cause the link selecton algorithm to rerun.

The link selection algorithm works as follows:

Starting from the lowest priority grouping value, assess all links
in that priority group.

The current priority-group is considered failed if:

o	"exclusive" NCUs exist and none are offline*/online,
o	"shared" NCUs exist and none are offline*/online,
o	"all" NCUs exist and all are not offline*/online,
o	no NCUs are offline*/online.

We do not invalidate a link that is offline* since its configuration
is in progress. This has the unfortunate side-effect that
wired links that do not do DL_NOTE_LINK_UP/DOWN will never
fail. If such links wish to be skipped, their priority group value
should be increased (prioritizing wireless links).

One a priority group has been selected, all links in groups above
_and_ below it need to be moved offline.

Location Activation
===================
A basic set of system-supplied locations are supplied - NoNet and
Automatic.  nwamd will apply the NoNet location until such a time
as an interface NCU is online, at which point it will switch
to the Automatic location.  If a user-supplied location is supplied,
and it is either manually enabled or its conditions are satisfied, it
will be preferred and activated instead.  Only one location can be
active at once since each location has its own specification of nameservices
etc.

ENM Activation
==============
ENMs are either manual or conditional in activation and will be
activated if they are enabled (manual) or if the conditions
are met (conditional).  Multiple ENMs can be active at once.