.\" "
.\" " The contents of this file are subject to the terms of the
.\" " Common Development and Distribution License (the "License").
.\" " You may not use this file except in compliance with the License.
.\" "
.\" " You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
.\" " or http://www.opensolaris.org/os/licensing.
.\" " See the License for the specific language governing permissions
.\" " and limitations under the License.
.\" "
.\" " When distributing Covered Code, include this CDDL HEADER in each
.\" " file and include the License file at usr/src/OPENSOLARIS.LICENSE.
.\" " If applicable, add the following below this CDDL HEADER, with the
.\" " fields enclosed by brackets "[]" replaced with your own identifying
.\" " information: Portions Copyright [yyyy] [name of copyright owner]
.\" "
.\" " CDDL HEADER END
.\" "
.\" "Copyright (c) 1999, 2010, Oracle and/or its affiliates. All rights reserved.
.\" "
.TH nightly 1 "6 July 2010"
.SH NAME
.I nightly
\- build an OS-Net consolidation overnight
.SH SYNOPSIS
\fBnightly [-in] [-V VERS] <env_file>\fP
.LP
.SH DESCRIPTION
.IX "OS-Net build tools" "nightly" "" "\fBnightly\fP"
.LP
.I nightly,
the mother of all build scripts,
can bringover, build, archive, package, error check, and
generally do everything it takes to
turn OS/Net consolidation source code into useful stuff.
It is customizable to permit you to run anything from a
simple build to all of the cross-checking a gatekeeper
needs.  The advantage to using
.I nightly
is that you build things correctly, consistently and
automatically, with the best practices; building with
.I nightly
can mean never having to say you're sorry to your
gatekeeper.
.LP
More
specifically,
.I nightly
performs the following tasks, in order, if
all these things are desired:
.LP
.RS
.TP
\(bu
perform a "make clobber" to clean up old binaries
.TP
\(bu
bringover from the identified parent gate/clone
.TP
\(bu
perform non-DEBUG and DEBUG builds
.TP
\(bu
list proto area files and compare with previous list
.TP
\(bu
copy updated proto area to parent
.TP
\(bu
list shared lib interface and compare with previous list
.TP
\(bu
perform a "make lint" of the kernel and report errors
.TP
\(bu
perform a "make check" to report hdrchk/cstyle errors
.TP
\(bu
report the presence of any core files
.TP
\(bu
check the ELF runtime attributes of all dynamic objects
.TP
\(bu
check for unreferenced files
.TP
\(bu
report on which proto area objects have changed (since the last build)
.TP
\(bu
report the total build time
.TP
\(bu
save a detailed log file for reference
.TP
\(bu
mail the user a summary of the completed build
.RE
.LP
The actions of the script are almost completely determined by
the environment variables in the
.I env
file, the only necessary argument.  Ths only thing you really
need to use 
.I nightly
is an
.I env
file that does what you want.
.LP
Like most of the other build tools in usr/src/tools, this script tends
to change on a fairly regular basis; do not expect to be able to build
OS/Net with a version of nightly significantly older than your source
tree.  It has what is effectively a Consolidation Private relationship
to other build tools and with many parts of the OS/Net makefiles,
although it may also be used to build other consolidations.
.LP
.SH NIGHTLY_OPTIONS
The environment variable NIGHTLY_OPTIONS controls the actions
.I nightly
will take as it proceeds.
The -i, -n, +t and -V options may also be used from the command
line to control the actions without editing your environment file.
The -i and -n options complete the build more quickly by bypassing
some actions. If NIGHTLY_OPTIONS is not set, then "-Bmt" build
options will be used.

.B Basic action options
.TP 10
.B \-D
Do a build with DEBUG on (non-DEBUG is built by default)
.TP
.B \-F
Do _not_ do a non-DEBUG build (use with -D to get just a DEBUG build)
.TP
.B \-M
Do not run pmodes (safe file permission checker)
.TP
.B \-i
Do an incremental build, suppressing the "make clobber" that by
default removes all existing binaries and derived files.  From the
command line, -i also suppresses the lint pass and the cstyle/hdrchk
pass
.TP
.B \-n
Suppress the bringover so that the build will start immediately with
current source code
.TP
.B \-o
Do an "old style" (pre-S10) build using root privileges to set OWNER
and GROUP from the Makefiles.
.TP
.B \-p
Create packages for regular install
.TP
.B \-U
Update proto area in the parent workspace
.TP
.B \-u
Update the parent workspace with files generated by the build, as follows.
.RS
.TP
\(bu
Copy proto_list_${MACH} and friends to usr/src in the parent.
.TP
\(bu
When used with -f, build a usr/src/unrefmaster.out in
the parent by merging all the usr/src/unref-${MACH}.out files in the
parent.
.TP
\(bu
When used with -A or -r, copy the contents of the resulting
ELF-data.${MACH} directory to usr/src/ELF-data.${MACH} in the parent
workspace.
.RE
.TP
.B \-m
Send mail to $MAILTO at end of build
.TP
.B \-t
Build and use the tools in $SRC/tools (default setting).
.TP
.B \+t
Use the build tools in "$ONBLD_TOOLS/bin".

.LP
.B Code checking options
.TP 10
.B \-A
Check for ABI discrepancies in .so files.
It is only required for shared object developers when there is an
addition, deletion or change of interface in the .so files.
.TP
.B \-C
Check for cstyle/hdrchk errors
.TP
.B \-f
Check for unreferenced files.  Since the full workspace must be built
in order to accurately identify unreferenced files, -f is ignored for
incremental (-i) builds, or builds that do not include -l, and -p.
.TP
.B \-r
Check the ELF runtime attributes of all dynamic objects
.TP
.B \-l
Do "make lint" in $LINTDIRS (default: $SRC n)
.TP
.B \-N
Do not run protocmp or checkpaths (note: this option is not
recommended, especially in conjunction with the \-p option)
.TP
.B \-W
Do not report warnings (for freeware gate ONLY)
.TP
.B \-w
Report which proto area objects differ between this and the last build.
See wsdiff(1) for details. Note that the proto areas used for comparison
are the last ones constructed as part of the build. As an example, if both
a non-debug and debug build are performed (in that order), then the debug
proto area will be used for comparison (which might not be what you want).
.LP
.B Groups of options
.TP 10
.B \-G
Gate keeper default group of options (-u)
.TP
.B \-I
Integration engineer default group of options (-mpu)
.TP
.B \-R
Default group of options for building a release (-mp)

.LP
.B Source Build options
.TP 10
.B \-S E | D | H
Build the Export, Domestic, or Hybrid source product. Only Export and
Domestic are truly buildable at this time.
.TP 10
.B \-S O
Simulate an OpenSolaris build on a full tree.  This can be used by
internal developers to ensure that they haven't broken the build for
external developers.
.LP
Source build options only make sense for a full internal tree (open
and closed source).  Only one source build option can be specified at
a time.

.LP
.B Miscellaneous options
.TP 10
.B \-O
generate deliverables for OpenSolaris.  Tarballs containing signed
cryptographic binaries and binaries
of closed-source components are put in $CODEMGR_WS.  (The
cryptographic tarballs are copies of the
ones that are put in the parent directory of
$PKGARCHIVE.)
.TP 10
.B \-V VERS
set the build version string to VERS, overriding VERSION
.TP
.B \-X
Copies the proto area and packages from the IHV and IHV-bin gates into the
nightly proto and package areas.  This is only available on i386.  See
.B REALMODE ENVIRONMENT VARIABLES
and
.B BUILDING THE IHV WORKSPACE
below.

.LP
.SH ENVIRONMENT VARIABLES
.LP
Here is a list of prominent environment variables that 
.I nightly
references and the meaning of each variable.
.LP
.RE
.B CODEMGR_WS
.RS 5
The root of your workspace, including whatever metadata is kept by
the source code management system.  This is the workspace in which the
build will be done. 
.RE
.LP
.B PARENT_WS
.RS 5
The root of the workspace that is the parent of the
one being built.  This is particularly relevant for configurations
with a main
workspace and build workspaces underneath it; see the 
\-u and \-U
options as well as the PKGARCHIVE environment variable, for more
information.
.RE
.LP
.B BRINGOVER_WS
.RS 5
This is the workspace from which
.I nightly
will fetch sources to either populate or update your workspace;
it defaults to $CLONE_WS.
.RE
.LP
.B CLOSED_BRINGOVER_WS
.RS 5
A full Mercurial workspace has two repositories: one for open source
and one for closed source.  If this variable is non-null, 
.I nightly
will pull from the repository that it names to get the closed source.
It defaults to $CLOSED_CLONE_WS.
.LP
If $CODEMGR_WS already exists and contains only the open repository,
.I nightly
will ignore this variable; you'll need to pull the closed repository
by hand if you want it.
.RE
.LP
.B CLONE_WS
.RS 5
This is the workspace from which
.I nightly
will fetch sources by default.  This is
often distinct from the parent, particularly if the parent is a gate.
.RE
.LP
.B CLOSED_CLONE_WS
.RS 5
This is the default closed-source Mercurial repository that 
.I nightly
might pull from (see
.B CLOSED_BRINGOVER_WS
for details).
.RE
.LP
.B SRC
.RS 5
Root of OS-Net source code, referenced by the Makefiles.  It is
the starting point of build activity.  It should be expressed
in terms of $CODEMGR_WS.
.RE
.LP
.B ROOT
.RS 5
Root of the proto area for the build.  The makefiles direct
installation of build products to this area and
direct references to these files by builds of commands and other
targets.  It should be expressed in terms of $CODEMGR_WS.
.LP
If $MULTI_PROTO is "no", $ROOT may contain a DEBUG or non-DEBUG
build.  If $MULTI_PROTO is "yes", $ROOT contains the DEBUG build and
$ROOT-nd contains the non-DEBUG build.
.LP
For OpenSolaris deliveries (\fB\-O\fR), $ROOT-closed contains a parallel
proto area containing the DEBUG build of just usr/closed components, and
$ROOT-nd-closed contains the non-DEBUG equivalent.
.RE
.LP
.B TOOLS_ROOT
.RS 5
Root of the tools proto area for the build.  The makefiles direct
installation of tools build products to this area.  Unless \fB+t\fR
is part of $NIGHTLY_OPTIONS, these tools will be used during the
build.
.LP
As built by nightly, this will always contain non-DEBUG objects.
Therefore, this will always have a -nd suffix, regardless of
$MULTI_PROTO.
.RE
.LP
.B MACH
.RS 5
The instruction set architecture of the build machine as given
by \fIuname -p\fP, e.g. sparc, i386.
.RE
.LP
.B LOCKNAME
.RS 5
The name of the file used to lock out multiple runs of
.IR nightly .
This should generally be left to the default setting.
.RE
.LP
.B ATLOG
.RS 5
The location of the log directory maintained by
.IR nightly .
This should generally be left to the default setting.
.RE
.LP
.B LOGFILE
.RS 5
The name of the log file in the $ATLOG directory maintained by
.IR nightly .
This should generally be left to the default setting.
.RE
.LP
.B STAFFER
.RS 5
The non-root account to use on the build machine for the
bringover from the clone or parent workspace.
This may not be the same identify used by the SCM.
.RE
.LP
.B MAILTO
.RS 5
The address to be used to send completion e-mail at the end of
the build (for the \-m option).
.RE
.LP
.B REF_PROTO_LIST
.RS 5
Name of file used with protocmp to compare proto area contents.
.RE
.LP
.B PARENT_ROOT
.RS 5
The parent root, which is the destination for copying the proto
area(s) when using the \-U option.
.RE
.LP
.B PARENT_TOOLS_ROOT
.RS 5
The parent tools root, which is the destination for copying the tools
proto area when using the \-U option.
.RE
.LP
.B RELEASE
.RS 5
The release version number to be used; e.g., 5.10.1 (Note: this is set
in Makefile.master and should not normally be overridden).
.RE
.LP
.B VERSION
.RS 5
The version text string to be used; e.g., "onnv:`date '+%Y-%m-%d'`".
.RE
.LP
.B RELEASE_DATE
.RS 5
The release date text to be used; e.g., October 2009. If not set in
your environment file, then this text defaults to the output from
$(LC_ALL=C date +"%B %Y"); e.g., "October 2009".
.RE
.LP
.B INTERNAL_RELEASE_BUILD
.RS 5
See Makefile.master - but it mostly controls id strings. Generally,
let
.I nightly
set this for you.
.RE
.LP
.B RELEASE_BUILD
.RS 5
Define this to build a release with a non-DEBUG kernel. 
Generally, let
.I nightly
set this for you based on its options.
.RE
.LP
.B PKGARCHIVE
.RS 5
The destination for packages.  This may be relative to
$CODEMGR_WS for private packages or relative to $PARENT_WS
if you have different workspaces for different architectures
but want one hierarchy of packages.
.RE
.LP
.B MAKEFLAGS
.RS 5
Set default flags to make; e.g., -k to build all targets regardless of errors.
.RE
.LP
.B UT_NO_USAGE_TRACKING
.RS 5
Disables usage reporting by listed Devpro tools. Otherwise it sends mail
to some Devpro machine every time the tools are used.
.RE
.LP
.B LINTDIRS
.RS 5
Directories to lint with the \-l option.
.RE
.LP
.B BUILD_TOOLS
.RS 5
BUILD_TOOLS is the root of all tools including the compilers; e.g.,
/ws/onnv-tools.  It is used by the makefile system, but not nightly.
.RE
.LP
.B ONBLD_TOOLS
.RS 5
ONBLD_TOOLS is the root of all the tools that are part of SUNWonbld; e.g.,
/ws/onnv-tools/onbld.  By default, it is derived from
.BR BUILD_TOOLS .
It is used by the makefile system, but not nightly.
.RE
.LP
.B SPRO_ROOT
.RS 5
The gate-defined default location for the Sun compilers, e.g.
/ws/onnv-tools/SUNWspro.  By default, it is derived from
.BR BUILD_TOOLS .
It is used by the makefile system, but not nightly.
.RE
.LP
.B JAVA_ROOT
.RS 5
The location for the java compilers for the build, generally /usr/java.
.RE
.LP
.B OPTHOME
.RS 5
The gate-defined default location of things formerly in /opt; e.g.,
/ws/onnv-tools.  This is used by nightly, but not the makefiles.
.RE
.LP
.B TEAMWARE
.RS 5
The gate-defined default location for the Teamware tools; e.g.,
/ws/onnv-tools/SUNWspro.  By default, it is derived from
.BR OPTHOME .
This is used by nightly, but not the makefiles.  There is no
corresponding variable for Mercurial or Subversion, which are assumed
to be installed in the default path.
.RE
.LP
.B EXPORT_SRC
.RS 5
The source product has no SCCS history, and is modified to remove source
that cannot be shipped. EXPORT_SRC is where the clear files are copied, then
modified with 'make EXPORT_SRC'.
.RE
.LP
.B CRYPT_SRC
.RS 5
CRYPT_SRC is similar to EXPORT_SRC, but after 'make CRYPT_SRC' the files in
xmod/cry_files are saved. They are dropped on the exportable source to create
the domestic build.
.LP
.RE
.B OPEN_SRCDIR
.RS 5
The open source tree is copied to this directory when simulating an
OpenSolaris build (\fB\-S O\fR).  It defaults to $CODEMGR_WS/open_src.
.LP
.RE
.B ON_CLOSED_BINS
.RS 5
OpenSolaris builds do not contain the closed source tree.  Instead,
the developer downloads a closed binaries tree and unpacks it.  
.B ON_CLOSED_BINS
tells nightly
where to find these closed binaries, so that it can add them into the
build.
.LP
.RE
.B ON_CRYPTO_BINS
.RS 5
This is the path to a compressed tarball that contains debug
cryptographic binaries that have been signed to allow execution
outside of Sun, e.g., $PARENT_WS/packages/$MACH/on-crypto.$MACH.bz2.
.I nightly
will automatically adjust the path for non-debug builds.  This tarball
is needed if the closed-source tree is not present.  Also, it is
usually needed when generating OpenSolaris deliverables from a project
workspace.  This is because most projects do not have access to the
necessary key and certificate that would let them sign their own
cryptographic binaries.
.LP
.RE
.B CHECK_PATHS
.RS 5
Normally, nightly runs the 'checkpaths' script to check for
discrepancies among the files that list paths to other files, such as
exception lists and req.flg.  Set this flag to 'n' to disable this
check, which appears in the nightly output as "Check lists of files."
.RE
.LP
.B CHECK_DMAKE
.RS 5
Nightly validates that the version of dmake encountered is known to be
safe to use.  Set this flag to 'n' to disable this test, allowing any
version of dmake to be used.
.RE
.LP
.B MULTI_PROTO
.RS 5
If "no" (the default), 
.I nightly
will reuse $ROOT for both the DEBUG and non-DEBUG builds.  If "yes",
the DEBUG build will go in $ROOT and the non-DEBUG build will go in
$ROOT-nd.  Other values will be treated as "no".  Use of the
.B \-O
flag forces MULTI_PROTO to "yes".
.RE
.LP
.SH NIGHTLY HOOK ENVIRONMENT VARIABLES
.LP
Several optional environment variables may specify commands to run at
various points during the build.  Commands specified in the hook
variable will be run in a subshell; command output will be appended to
the mail message and log file.  If the hook exits with a non-zero
status, the build is aborted immediately.  Environment variables
defined in the environment file will be available.
.LP
.B SYS_PRE_NIGHTLY
.RS 5
Run just after the workspace lock is acquired.  This is reserved for
per-build-machine customizations and should be set only in /etc/nightly.conf
.RE
.LP
.B PRE_NIGHTLY
.RS 5
Run just after SYS_PRE_NIGHTLY.
.RE
.LP
.B PRE_BRINGOVER
.RS 5
Run just before bringover is started; not run if no bringover is done.
.RE
.LP
.B POST_BRINGOVER
.RS 5
Run just after bringover completes; not run if no bringover is done.
.RE
.LP
.B POST_NIGHTLY
.RS 5
Run after the build completes, with the return status of nightly - one
of "Completed", "Interrupted", or "Failed" - available in the
environment variable NIGHTLY_STATUS.
.RE
.LP
.B SYS_POST_NIGHTLY
.RS 5
This is reserved for per-build-machine customizations, and runs
immedately after POST_NIGHTLY.
.RE
.LP
.SH REALMODE ENVIRONMENT VARIABLES
.LP
The following environment variables referenced by
.I nightly
are only required when the -X option is used.
.LP
.RE
.B IA32_IHV_WS
.RS 5
Reference to the IHV workspace containing IHV driver binaries.
The IHV workspace must be fully built before starting the ON realmode build.
.LP
.RE
.B IA32_IHV_ROOT
.RS 5
Reference to the IHV workspace proto area.
The IHV workspace must be fully built before starting the ON realmode build.
.LP
.RE
.B IA32_IHV_PKGS
.RS 5
Reference to the IHV workspace packages.  If this is empty or the directory
is non-existent, then nightly will skip copying the packages.
.LP
.RE
.B IA32_IHV_BINARY_PKGS
.RS 5
Reference to binary-only IHV packages.  If this is empty or the directory
is non-existent, then nightly will skip copying the packages.
.LP
.RE
.B SPARC_RM_PKGARCHIVE
.RS 5
Destination for sparc realmode package SUNWrmodu.
Yes, this sparc package really is built on x86.
.SH FILES
.LP
.RS 5
/etc/nightly.conf
.RE
.LP
If present, nightly executes this file just prior to executing the
.I env
file.  
.SH BUILDING THE IHV WORKSPACE
.LP
The IHV workspace can be built with
.I nightly.
The recommended options are:
.LP
.RS 5
NIGHTLY_OPTIONS="-pmWN"
.RE
.LP
None of the realmode environment variables needed for ON realmode builds
are required to build the IHV workspace.
.SH EXAMPLES
.LP
Start with the example file in usr/src/tools/env/developer.sh
(or gatekeeper.sh), copy to myenv and make your changes.
.LP
.PD 0
# grep NIGHTLY_OPTIONS myenv
.LP
NIGHTLY_OPTIONS="-ACrlapDm"
.LP
export NIGHTLY_OPTIONS
.LP
# /opt/onbld/bin/nightly -i myenv
.PD
.LP
.SH SEE ALSO
.BR bldenv (1)