xref: /freebsd/contrib/bmake/mk/mk-files.txt (revision cccdaf507eee8fb34494b4624eb85bb951e323c8)
1mk-files
2********
3
4The term ``mk-files`` refers to a collection of ``*.mk`` files.
5
6You need bmake_ or a *recent* NetBSD_ make.
7If in doubt use bmake_.
8
9Introduction
10============
11
12Many years ago, when building large software projects, I used GNU make
13(or my own patched version of it), and had developed a set of macros
14to simplify developing complex build trees.
15
16Since the early 90's my main development machines, run BSD
17(NetBSD_ to be precise, and more recently FreeBSD), and the BSD source
18tree is good example of a large software project.
19It quickly became clear that ``/usr/share/mk/*.mk`` were a great
20model, but at the time were quite tightly linked to building the BSD tree.
21
22Much as I liked using NetBSD, my customers were more likely to be
23using SunOS, HP-UX etc, so I started on bmake_ and a portable collection
24of mk-files (mk.tar.gz_).  NetBSD provided much of the original structure.
25
26Since then I've added a lot of features to NetBSD's make and hence to
27bmake which is kept closely in sync.  The mk-files however have
28diverged quite a bit, though ideas are still picked up from NetBSD
29and FreeBSD.
30
31Basics
32------
33
34The BSD build model is very simple.  A directory produces one
35component, which is generally either a library or a program.
36Library makefiles include ``lib.mk`` and programs include ``prog.mk``
37and they *do the right thing*.
38
39A simple library makefile might look like::
40
41	LIB = sig
42
43	SRCS = \
44		sigaction.c \
45		sigcompat.c \
46		sighdl.c
47
48	.include <lib.mk>
49
50a simple program makefile::
51
52	PROG = cat
53
54	SRCS = cat.c
55
56	.include <prog.mk>
57
58in such cases even the ``SRCS`` line is unnecessary as ``prog.mk``
59will default it to ``${PROG}.c``.
60
61It is the sensible use of defaults and the plethora of macro modifiers
62provided by bmake_ that allow simple makefiles such as the above to
63*just work* on many different systems.
64
65
66mk-files
67========
68
69This section provides a brief description of some of the ``*.mk``
70files.
71
72sys.mk
73------
74
75When bmake starts, it looks for ``sys.mk`` and reads it before doing
76anything else.  Thus, this is the place to setup the environment for
77everyone else.
78
79In this distribution, ``sys.mk`` avoids doing anything platform or
80site dependent.
81It is quite short, and includes a number of other files (which may or
82may not exists)
83
84sys.env.mk
85	If it exists, is expected to do things like conditioning the
86	environment.  Since it will only be included by the initial
87	instance of bmake, it should ``.export`` anything that
88	sub-makes might need.
89
90examples/sys.clean-env.mk
91	An example of how to clean the environment.
92	See the file for all the details::
93
94		.if ${MAKE_VERSION} >= 20100606 && ${.MAKE.LEVEL} == 0
95		# we save any env var that starts with these
96		MAKE_SAVE_ENV_PREFIX += SB MK MAKE MACHINE NEED_ CCACHE DISTCC USE_ SSH
97		MAKE_SAVE_ENV_VARS += \
98			PATH HOME USER LOGNAME \
99			SRCTOP OBJTOP OBJROOT \
100			${_env_vars}
101
102		_env_vars != env | egrep '^(${MAKE_SAVE_ENV_PREFIX:ts|})' | sed 's,=.*,,'; echo
103		_export_list =
104		.for v in ${MAKE_SAVE_ENV_VARS:O:u}
105		.if !empty($v)
106		_export_list += $v
107		$v := ${$v}
108		.endif
109		.endfor
110		# now clobber the environment
111		.unexport-env
112
113		# list of vars that we handle specially below
114		_tricky_env_vars = MAKEOBJDIR
115		# export our selection - sans tricky ones
116		.export ${_export_list:${_tricky_env_vars:${M_ListToSkip}}}
117
118		# this next bit may need tweaking
119		.if defined(MAKEOBJDIR)
120		srctop := ${SRCTOP:U${SB_SRC:U${SB}/src}}
121		objroot := ${OBJROOT:U${SB_OBJROOT:U${SB}/${SB_OBJPREFIX}}}
122		# we'll take care of MACHINE below
123		objtop := ${OBJTOP:U${objroot}${MACHINE}}
124		.if !empty(objtop)
125		# we would normally want something like (/bin/sh):
126		# MAKEOBJDIR="\${.CURDIR:S,${SRCTOP},${OBJROOT}\${MACHINE},}"
127		# the $$ below is how we achieve the same result here.
128		# since everything saved from the environment above
129		# has run through := we need to compensate for ${MACHINE}
130		MAKEOBJDIR = $${.CURDIR:S,${srctop},${objtop:S,${MACHINE},\${MACHINE},},}
131
132		# export these as-is, and do not track...
133		.export-env ${_tricky_env_vars}
134		# now evaluate for ourselves
135		.for v in ${_tricky_env_vars}
136		$v := ${$v}
137		.endfor
138
139		.endif
140		.endif
141		.endif
142
143
144host-target.mk
145	Is used to set macros like ``HOST_TARGET``, ``HOST_OS`` and
146	``host_os`` which are used to find the next step.
147	Note: since 20130303 bmake provides ``.MAKE.OS`` set to
148	the equivalent of ``HOST_OS``.
149
150sys/\*.mk
151	Platform specific additions, such as ``Darwin.mk`` or ``SunOS.mk``
152	set things like ``HOST_LIBEXT = .dylib`` for Darwin or
153	``SHLIB_FULLVERSION = ${SHLIB_MAJOR}`` for SunOS 5.
154	If there is no OS specific file, ``sys/Generic.mk`` is used.
155
156local.sys.mk
157	Any ``local.*.mk`` file is not part of the distribution.
158	This provides a hook for sites to do extra setup without
159	having to edit the distributed files.
160
161
162The above arrangement makes it easy for the mk files to be part of a
163src tree on an NFS volume and to allow building on multiple platforms.
164
165options.mk
166----------
167
168Inspired by FreeBSD's ``bsd.own.mk`` but more flexible.
169FreeBSD now have similar functionality in ``bsd.mkopt.mk``.
170
171It allows users to express their intent with respect to options
172``MK_*`` by setting ``WITH_*`` or ``WITHOUT_*``.
173
174Note: ``WITHOUT_*`` wins if both are set, and makefiles can set
175``NO_*`` to say they cannot handle that option, or even ``MK_*`` if
176they really need to.
177
178lib.mk
179------
180
181This file is used to build a number of different libraries from the
182same SRCS.
183
184``lib${LIB}.a``
185	An archive lib of ``.o`` files, this is the default
186
187``lib${LIB}_p.a``
188	A profiled lib of ``.po`` files.
189	Still an archive lib, but all the objects are built with
190	profiling in mind - hence the different extension.
191	It is skipped if ``MK_PROFILE`` is "no".
192
193``lib${LIB}_pic.a``
194	An archive of ``.so`` objects compiled for relocation.
195	On NetBSD this is the input to ``lib${LIB}.${LD_so}``, it is
196	skipped if ``MK_PIC`` or ``MK_PICLIB`` are "no".
197
198``lib${LIB}.${LD_so}``
199	A shared library.  The value of ``LD_so`` is very platform
200	specific.  For example::
201
202		# SunOS 5 and most other ELF systems
203		libsslfd.so.1
204
205		# Darwin
206		libsslfd.1.dylib
207
208	This library will only be built if ``SHLIB_MAJOR`` has
209	a value, and ``MK_PIC`` is not set to "no".
210
211There is a lot of platform specific tweaking in ``lib.mk``, largely the
212result of the original distributions trying to avoid interfering with
213the system's ``sys.mk``.
214
215libnames.mk
216-----------
217
218This is included by both ``prog.mk`` and ``lib.mk`` and tries to
219include ``*.libnames.mk`` of which:
220
221``local.libnames.mk``
222	does not exist unless you create it.  It is a handy way for you
223	to customize without touching the distributed files.
224	For example, on a test machine I needed to build openssl but
225	not install it, so put the following in ``local.libnames.mk``::
226
227		.if ${host_os} == "sunos"
228		LIBCRYPTO = ${OBJTOP}/openssl/lib/crypto/libcrypto${DLIBEXT}
229		LIBSSL = ${OBJTOP}/openssl/lib/ssl/libssl${DLIBEXT}
230		INCLUDES_libcrypto = -I${OBJ_libcrypto}
231		.endif
232
233	The makefile created an openssl dir in ``${OBJ_libcrypto}`` to
234	gather all the headers. dpadd.mk_ did the rest.
235
236``host.libnames.mk``
237	contains logic to find any libs named in ``HOST_LIBS`` in
238	``HOST_LIBDIRS``.
239
240Each file above gets an opportunity to define things like::
241
242	LIBSSLFD	?= ${OBJTOP}/ssl/lib/sslfd/libsslfd${DLIBEXT}
243	INCLUDES_libsslfd = -I${SRC_libsslfd}/h -I${OBJ_libslfd}
244
245these are used by dpadd.mk_ and will be explained below.
246
247dpadd.mk
248--------
249
250This file looks like line noise, and is best considered read-only.
251However it provides some very useful functionality, which simplifies the build.
252
253Makefiles can use the LIB* macros defined via libnames.mk_ or anywhere
254else in various ways::
255
256	# indicate that we need to include headers from LIBCRYPTO
257	# this would result in ${INCLUDES_libcrypto} being added to CFLAGS.
258	SRC_LIBS += ${LIBCRYPTO}
259
260	# indicate that libsslfd must be built already.
261	# it also has the same effect as SRC_LIBS
262	DPADD += ${LIBSSLFD}
263
264	# indicate that not only must libsslfd be built,
265	# but that we need to link with it.
266	# this is almost exactly equivalent to
267	# DPADD += ${LIBSSLFD}
268	# LDADD += -L${LIBSSLFD:H} -lsslfd
269	# and mostly serves to ensure that DPADD and LDADD are in sync.
270	DPLIBS += ${LIBSSLFD}
271
272Any library (referenced by its full path) in any of the above, is
273added to ``DPMAGIC_LIBS`` with the following results, for each lib *foo*.
274
275``SRC_libfoo``
276	Is set to indicate where the src for libfoo is.
277	By default it is derived from ``LIBFOO`` by replacing
278	``${OBJTOP}`` with ``${SRCTOP}``.
279
280``OBJ_libfoo``
281	Not very exciting, is just the dir where libfoo lives.
282
283``INCLUDES_libfoo``
284	What to add to ``CFLAGS`` to find the public headers.
285	The default varies.  If ``${SRC_libfoo}/h`` exists, it is assumed
286	to be the home of all public headers and thus the default is
287	``-I${SRC_libfoo}/h``
288
289	Otherwise we make no assumptions and the default is
290	``-I${SRC_libfoo} -I${OBJ_libfoo}``
291
292``LDADD_libfoo``
293	This only applies to libs reference via ``DPLIBS``.
294	The default is ``-lfoo``, ``LDADD_*`` provides a hook to
295	instantiate other linker flags at the appropriate point
296	without losing the benfits of ``DPLIBS``.
297
298prog.mk
299-------
300
301Compiles the specified SRCS and links them and the nominated libraries
302into a program.  Prog makefiles usually need to list the libraries
303that need to be linked.   We prefer use of ``DPLIBS`` but the more
304traditional ``DPADD`` and ``LDADD`` work just as well.
305That is::
306
307	DPLIBS += ${LIBCRYPTO}
308
309is equivalent to::
310
311	DPADD += ${LIBCRYPTO}
312	LDADD += -lcrypto
313
314obj.mk
315------
316
317One of the cool aspects of BSD make, is its support for separating
318object files from the src tree.  This is also the source of much
319confusion for people unfamiliar with it.
320
321Traditionally one had to do a separate ``make obj`` pass through the
322tree.  If ``MK_AUTO_OBJ`` is set we include auto.obj.mk_.
323
324In fact if ``MKOBJDIRS`` is set to "auto", `sys.mk`_ will set
325``MK_AUTO_OBJ=yes`` and include auto.obj.mk_ since it is best done early.
326
327auto.obj.mk
328-----------
329
330Creates object dirs and leverages the ``.OBJDIR`` target introduced
331some years ago to NetBSD make, to use them.
332
333
334subdir.mk
335---------
336
337This is the traditional means of walking the tree.  A makefile sets
338``SUBDIR`` to the list of sub-dirs to visit.
339
340If ``SUBDIR_MUST_EXIST`` is set, missing directories cause an error,
341otherwise a warning is issued.  If you don't even want the warning,
342set ``MISSING_DIR=continue``.
343
344Traditionally, ``subdir.mk`` prints clues as it visits each subdir::
345
346	===> ssl
347	===> ssl/lib
348	===> ssl/lib/sslfd
349
350you can suppress that - or enhance it by setting ``ECHO_DIR``::
351
352	# suppress subdir noise
353	ECHO_DIR=:
354	# print time stamps
355	ECHO_DIR=echo @ `date "+%s [%Y-%m-%d %T] "`
356
357I prefer to use `dirdeps.mk`_ which makes ``subdir.mk`` irrelevant.
358
359links.mk
360--------
361
362Provides rules for processing lists of ``LINKS`` and ``SYMLINKS``.
363Each is expected to be a list of ``link`` and ``target`` pairs
364(``link`` -> ``target``).
365
366The logic is generally in a ``_*_SCRIPT`` which is referenced in a
367``_*_USE`` (``.USE``) target.
368
369The ``_BUILD_*`` forms are identical, but do not use ``${DESTDIR}``
370and so are useful for creating symlinks during the build phase.
371For example::
372
373	SYMLINKS += ${.CURDIR}/${MACHINE_ARCH}/include machine
374	header_links: _BUILD_SYMLINKS_USE
375
376	md.o: header_links
377
378would create a symlink called ``machine`` in ``${.OBJDIR}`` pointing to
379``${.CURDIR}/${MACHINE_ARCH}/include`` before compiling ``md.o``
380
381
382autoconf.mk
383-----------
384
385Deals with running (or generating) GNU autoconf ``configure`` scripts.
386
387dep.mk
388------
389
390Deals with collecting dependencies.  Another useful feature of BSD
391make is the separation of this sort of information into a ``.depend``
392file.  ``MKDEP_CMD`` needs to point to a suitable tool (like mkdeps.sh_)
393
394If ``MK_AUTODEP`` is "yes" it sets ``MKDEP_MK`` to autodep.mk_ by default.
395
396``MKDEP_MK`` can also be set to `auto.dep.mk`_ which is more efficient
397but does not support an explicit ``depend`` target.
398
399autodep.mk
400----------
401
402Leverages the ``-MD`` feature of recent GCC to collect dependency
403information as a side effect of compilation.  With this GCC puts
404dependency info into a ``.d`` file.
405
406Unfortunately GCC bases the name of the ``.d`` file on the name of the
407input rather than the output file, which causes problems when the same
408source is compiled different ways.  The latest GCC supports ``-MF`` to
409name the ``.d`` file and ``-MT`` to control the name to put as the
410dependent.
411
412Recent bmake allows dependencies for the ``.END`` target (run at the
413end if everything was successful), and ``autodep.mk`` uses this to
414post process the ``.d`` files into ``.depend``.
415
416auto.dep.mk
417-----------
418
419A much simpler implementation than autodep.mk_ it uses
420``-MF ${.TARGET:T}.d``
421to avoid possible conflicts during parallel builds.
422This precludes the use of suffix rules to drive ``make depend``, so
423dep.mk_ handles that if specifically requested.
424
425If ``bmake`` is 20160218 or newer, ``auto.dep.mk`` uses ``.dinclude``
426to includes the ``*.d`` files directly thus avoiding the need to
427create a ``.depend`` file from them.
428
429own.mk
430------
431
432Normally included by ``init.mk`` (included by ``lib.mk`` and
433``prog.mk`` etc), sets macros for default ownership  etc.
434
435It includes ``${MAKECONF}`` if it is defined and exists.
436
437ldorder.mk
438----------
439
440Leverages ``bmake`` to compute optimal link order for libraries.
441This works nicely and makes refactoring a breeze - so long as you
442have no (or few) cicular dependencies between libraries.
443
444Consider this experimental.
445
446man.mk
447------
448
449Deals with man pages.
450
451warnings.mk
452-----------
453
454This provides a means of fine grained control over warnings on a per
455``${MACHINE}`` or even file basis.
456
457A makefile sets ``WARNINGS_SET`` to name a list of warnings
458and individual ``W_*`` macros can be used to tweak them.
459For example::
460
461	WARNINGS_SET = HIGH
462	W_unused_sparc = -Wno-unused
463
464would add all the warnings in ``${HIGH_WARNINGS}`` to CFLAGS, but
465on sparc, ``-Wno-unused`` would replace ``-Wunused``.
466
467You should never need to edit ``warnings.mk``, it will include
468``warnings-sets.mk`` and/or ``local.warnings.mk`` to pick up
469customizations.
470
471rst2htm.mk
472----------
473
474Logic to simplify generating HTML (and PDF) documents from ReStructuredText.
475
476cython.mk
477---------
478
479Logic to build Python C interface modules using Cython_
480
481.. _Cython: http://www.cython.org/
482
483cc-wrap.mk
484----------
485
486This makefile leverages two new features in bmake 20220126 and later.
487
488First is the ablity to set target local variables (GNU make has done
489this for ages).
490
491The second (only intersting if using `meta mode`_)
492allows filtering commands before comparison with previous run to
493decide if a target is out-of-date.
494
495In the past, making use of compiler wrappers like ``ccache``,
496``distcc`` or the newer ``icecc`` could get quite ugly.
497Using ``cc-wrap.mk`` it could not be simpler.
498
499jobs.mk
500-------
501
502This should be included by the top-level makefile.
503If you do::
504
505	make something-jobs
506
507then ``jobs.mk`` will run::
508
509	make -j${JOB_MAX} someting > ${JOB_LOGDIR}/something.log 2>&1
510
511this ensures you get a build log and JOB_MAX is assumed to be set
512optimally for the host.
513
514META_MODE
515=========
516
517The 20110505 and later versions of ``mk-files`` include a number of
518makefiles contributed by Juniper Networks, Inc.
519These allow the latest version of bmake_ to run in `meta mode`_
520see `dirdeps.mk`_ and DIRDEPS_BUILD_ below.
521
522.. _`dirdeps.mk`: /help/sjg/dirdeps.htm
523.. _`meta mode`: bmake-meta-mode.htm
524
525DIRDEPS_BUILD
526=============
527
528When the `meta mode`_ was originally done, there was no distinction
529between META_MODE_ and ``DIRDEPS_BUILD``, but as these were integrated
530into FreeBSD it became clear that META_MODE_ could be useful to many
531developers independently of ``DIRDEPS_BUILD``.
532
533Thus today we distinguish between the two.
534We have the following makefiles which are relevant to
535``DIRDEPS_BUILD`` or META_MODE_::
536
537	share/mk/auto.obj.mk
538	share/mk/dirdeps-cache-update.mk
539	share/mk/dirdeps-only.mk
540	share/mk/dirdeps-options.mk
541	share/mk/dirdeps-targets.mk
542	share/mk/dirdeps.mk
543	share/mk/gendirdeps.mk
544	share/mk/host-target.mk
545	share/mk/install-new.mk
546	share/mk/meta.autodep.mk
547	share/mk/meta.stage.mk
548	share/mk/meta.sys.mk
549	share/mk/meta2deps.py
550	share/mk/meta2deps.sh
551	share/mk/sys.dependfile.mk
552	share/mk/sys.dirdeps.mk
553
554and the following are typically used for customization.
555See `freebsd-meta-mode`_ and `netbsd-meta-mode`_::
556
557	share/mk/local.dirdeps-build.mk
558	share/mk/local.dirdeps-missing.mk
559	share/mk/local.dirdeps.mk
560	share/mk/local.meta.sys.mk
561	share/mk/local.sys.dirdeps.env.mk
562	share/mk/local.sys.dirdeps.mk
563	share/mk/local.sys.mk
564
565
566Install
567=======
568
569You can use the content of mk.tar.gz_ without installing at all.
570
571The script ``install-mk`` takes care of copying ``*.mk`` into a
572destination directory, and unless told not to, create ``bsd.*.mk`` links
573for ``lib.mk`` etc.
574
575If you just want to create the ``bsd.*.mk`` links in the directory
576where you unpacked the tar file, you can::
577
578	./mk/install-mk ./mk
579
580------
581
582.. _bmake: bmake.htm
583.. _NetBSD: http://www.netbsd.org/
584.. _mkdeps.sh: https://www.crufty.net/ftp/pub/sjg/mkdeps.sh
585.. _mk.tar.gz: https://www.crufty.net/ftp/pub/sjg/mk.tar.gz
586.. _`freebsd-meta-mode`: https://www.crufty.net/sjg/docs/freebsd-meta-mode.htm
587.. _`netbsd-meta-mode`: https://www.crufty.net/sjg/docs/netbsd-meta-mode.htm
588
589:Author: sjg@crufty.net
590:Revision: $Id: mk-files.txt,v 1.23 2023/05/11 22:55:08 sjg Exp $
591:Copyright: Crufty.NET
592