1mk-files 2******** 3 4The term ``mk-files`` refers to a collection of ``*.mk`` files. 5 6You need bmake_ or a *recent* NetBSD_ make. 7If in doubt use bmake_. 8 9Introduction 10============ 11 12Many years ago, when building large software projects, I used GNU make 13(or my own patched version of it), and had developed a set of macros 14to simplify developing complex build trees. 15 16Since the early 90's my main development machines, run BSD 17(NetBSD_ to be precise, and more recently FreeBSD), and the BSD source 18tree is good example of a large software project. 19It quickly became clear that ``/usr/share/mk/*.mk`` were a great 20model, but at the time were quite tightly linked to building the BSD tree. 21 22Much as I liked using NetBSD, my customers were more likely to be 23using SunOS, HP-UX etc, so I started on bmake_ and a portable collection 24of mk-files (mk.tar.gz_). NetBSD provided much of the original structure. 25 26Since then I've added a lot of features to NetBSD's make and hence to 27bmake which is kept closely in sync. The mk-files however have 28diverged quite a bit, though ideas are still picked up from NetBSD 29and FreeBSD. 30 31Basics 32------ 33 34The BSD build model is very simple. A directory produces one 35component, which is generally either a library or a program. 36Library makefiles include ``lib.mk`` and programs include ``prog.mk`` 37and they *do the right thing*. 38 39A simple library makefile might look like:: 40 41 LIB = sig 42 43 SRCS = \ 44 sigaction.c \ 45 sigcompat.c \ 46 sighdl.c 47 48 .include <lib.mk> 49 50a simple program makefile:: 51 52 PROG = cat 53 54 SRCS = cat.c 55 56 .include <prog.mk> 57 58in such cases even the ``SRCS`` line is unnecessary as ``prog.mk`` 59will default it to ``${PROG}.c``. 60 61It is the sensible use of defaults and the plethora of macro modifiers 62provided by bmake_ that allow simple makefiles such as the above to 63*just work* on many different systems. 64 65 66mk-files 67======== 68 69This section provides a brief description of some of the ``*.mk`` 70files. 71 72The makefiles ``lib.mk``, ``prog.mk``, ``init.mk``, ``own.mk``, 73``dep.mk`` and ``man.mk`` are more or less equivalent to ``bsd.*.mk`` 74found in BSD, and when installed on non-BSD platforms get symlinked as 75``bsd.*.mk`` as well. 76 77The other makefiles (apart from ``sys.mk``) can be used in conjunction 78with ``bsd.*.mk`` on BSD. 79 80sys.mk 81------ 82 83When bmake starts, it looks for ``sys.mk`` and reads it before doing 84anything else. Thus, this is the place to setup the environment for 85everyone else. 86 87In this distribution, ``sys.mk`` avoids doing anything platform or 88site dependent. 89It is quite short, and includes a number of other files (which may or 90may not exists) 91 92sys.env.mk 93 If it exists, is expected to do things like conditioning the 94 environment. Since it will only be included by the initial 95 instance of bmake, it should ``.export`` anything that 96 sub-makes might need. 97 98examples/sys.clean-env.mk 99 An example of how to clean the environment. 100 See the file for all the details:: 101 102 .if ${MAKE_VERSION} >= 20100606 && ${.MAKE.LEVEL} == 0 103 # we save any env var that starts with these 104 MAKE_SAVE_ENV_PREFIX += SB MK MAKE MACHINE NEED_ CCACHE DISTCC USE_ SSH 105 MAKE_SAVE_ENV_VARS += \ 106 PATH HOME USER LOGNAME \ 107 SRCTOP OBJTOP OBJROOT \ 108 ${_env_vars} 109 110 _env_vars != env | egrep '^(${MAKE_SAVE_ENV_PREFIX:ts|})' | sed 's,=.*,,'; echo 111 _export_list = 112 .for v in ${MAKE_SAVE_ENV_VARS:O:u} 113 .if !empty($v) 114 _export_list += $v 115 $v := ${$v} 116 .endif 117 .endfor 118 # now clobber the environment 119 .unexport-env 120 121 # list of vars that we handle specially below 122 _tricky_env_vars = MAKEOBJDIR 123 # export our selection - sans tricky ones 124 .export ${_export_list:${_tricky_env_vars:${M_ListToSkip}}} 125 126 # this next bit may need tweaking 127 .if defined(MAKEOBJDIR) 128 srctop := ${SRCTOP:U${SB_SRC:U${SB}/src}} 129 objroot := ${OBJROOT:U${SB_OBJROOT:U${SB}/${SB_OBJPREFIX}}} 130 # we'll take care of MACHINE below 131 objtop := ${OBJTOP:U${objroot}${MACHINE}} 132 .if !empty(objtop) 133 # we would normally want something like (/bin/sh): 134 # MAKEOBJDIR="\${.CURDIR:S,${SRCTOP},${OBJROOT}\${MACHINE},}" 135 # the $$ below is how we achieve the same result here. 136 # since everything saved from the environment above 137 # has run through := we need to compensate for ${MACHINE} 138 MAKEOBJDIR = $${.CURDIR:S,${srctop},${objtop:S,${MACHINE},\${MACHINE},},} 139 140 # export these as-is, and do not track... 141 .export-env ${_tricky_env_vars} 142 # now evaluate for ourselves 143 .for v in ${_tricky_env_vars} 144 $v := ${$v} 145 .endfor 146 147 .endif 148 .endif 149 .endif 150 151 152host-target.mk 153 Is used to set macros like ``HOST_TARGET``, ``HOST_OS`` and 154 ``host_os`` which are used to find the next step. 155 Note: since 20130303 bmake provides ``.MAKE.OS`` set to 156 the equivalent of ``HOST_OS``. 157 158sys/\*.mk 159 Platform specific additions, such as ``Darwin.mk`` or ``SunOS.mk`` 160 set things like ``HOST_LIBEXT = .dylib`` for Darwin or 161 ``SHLIB_FULLVERSION = ${SHLIB_MAJOR}`` for SunOS 5. 162 If there is no OS specific file, ``sys/Generic.mk`` is used. 163 164local.sys.mk 165 Any ``local.*.mk`` file is not part of the distribution. 166 This provides a hook for sites to do extra setup without 167 having to edit the distributed files. 168 169 170The above arrangement makes it easy for the mk files to be part of a 171src tree on an NFS volume and to allow building on multiple platforms. 172 173options.mk 174---------- 175 176Inspired by FreeBSD's ``bsd.own.mk`` but more flexible. 177FreeBSD now have similar functionality in ``bsd.mkopt.mk``. 178 179It allows users to express their intent with respect to options 180``MK_*`` by setting ``WITH_*`` or ``WITHOUT_*``. 181 182Note: ``WITHOUT_*`` wins if both are set, and makefiles can set 183``NO_*`` to say they cannot handle that option, or even ``MK_*`` if 184they really need to. 185 186lib.mk 187------ 188 189This file is used to build a number of different libraries from the 190same SRCS. 191 192``lib${LIB}.a`` 193 An archive lib of ``.o`` files, this is the default 194 195``lib${LIB}_p.a`` 196 A profiled lib of ``.po`` files. 197 Still an archive lib, but all the objects are built with 198 profiling in mind - hence the different extension. 199 It is skipped if ``MK_PROFILE`` is "no". 200 201``lib${LIB}_pic.a`` 202 An archive of ``.so`` objects compiled for relocation. 203 On NetBSD this is the input to ``lib${LIB}.${LD_so}``, it is 204 skipped if ``MK_PIC`` or ``MK_PICLIB`` are "no". 205 206``lib${LIB}.${LD_so}`` 207 A shared library. The value of ``LD_so`` is very platform 208 specific. For example:: 209 210 # SunOS 5 and most other ELF systems 211 libsslfd.so.1 212 213 # Darwin 214 libsslfd.1.dylib 215 216 This library will only be built if ``SHLIB_MAJOR`` has 217 a value, and ``MK_PIC`` is not set to "no". 218 219There is a lot of platform specific tweaking in ``lib.mk``, largely the 220result of the original distributions trying to avoid interfering with 221the system's ``sys.mk``. 222 223libnames.mk 224----------- 225 226This is included by both ``prog.mk`` and ``lib.mk`` and tries to 227include ``*.libnames.mk`` of which: 228 229``local.libnames.mk`` 230 does not exist unless you create it. It is a handy way for you 231 to customize without touching the distributed files. 232 For example, on a test machine I needed to build openssl but 233 not install it, so put the following in ``local.libnames.mk``:: 234 235 .if ${host_os} == "sunos" 236 LIBCRYPTO = ${OBJTOP}/openssl/lib/crypto/libcrypto${DLIBEXT} 237 LIBSSL = ${OBJTOP}/openssl/lib/ssl/libssl${DLIBEXT} 238 INCLUDES_libcrypto = -I${OBJ_libcrypto} 239 .endif 240 241 The makefile created an openssl dir in ``${OBJ_libcrypto}`` to 242 gather all the headers. dpadd.mk_ did the rest. 243 244``host.libnames.mk`` 245 contains logic to find any libs named in ``HOST_LIBS`` in 246 ``HOST_LIBDIRS``. 247 248Each file above gets an opportunity to define things like:: 249 250 LIBSSLFD ?= ${OBJTOP}/ssl/lib/sslfd/libsslfd${DLIBEXT} 251 INCLUDES_libsslfd = -I${SRC_libsslfd}/h -I${OBJ_libslfd} 252 253these are used by dpadd.mk_ and will be explained below. 254 255dpadd.mk 256-------- 257 258This file looks like line noise, and is best considered read-only. 259However it provides some very useful functionality, which simplifies the build. 260 261Makefiles can use the LIB* macros defined via libnames.mk_ or anywhere 262else in various ways:: 263 264 # indicate that we need to include headers from LIBCRYPTO 265 # this would result in ${INCLUDES_libcrypto} being added to CFLAGS. 266 SRC_LIBS += ${LIBCRYPTO} 267 268 # indicate that libsslfd must be built already. 269 # it also has the same effect as SRC_LIBS 270 DPADD += ${LIBSSLFD} 271 272 # indicate that not only must libsslfd be built, 273 # but that we need to link with it. 274 # this is almost exactly equivalent to 275 # DPADD += ${LIBSSLFD} 276 # LDADD += -L${LIBSSLFD:H} -lsslfd 277 # and mostly serves to ensure that DPADD and LDADD are in sync. 278 DPLIBS += ${LIBSSLFD} 279 280Any library (referenced by its full path) in any of the above, is 281added to ``DPMAGIC_LIBS`` with the following results, for each lib *foo*. 282 283``SRC_libfoo`` 284 Is set to indicate where the src for libfoo is. 285 By default it is derived from ``LIBFOO`` by replacing 286 ``${OBJTOP}`` with ``${SRCTOP}``. 287 288``OBJ_libfoo`` 289 Not very exciting, is just the dir where libfoo lives. 290 291``INCLUDES_libfoo`` 292 What to add to ``CFLAGS`` to find the public headers. 293 The default varies. If ``${SRC_libfoo}/h`` exists, it is assumed 294 to be the home of all public headers and thus the default is 295 ``-I${SRC_libfoo}/h`` 296 297 Otherwise we make no assumptions and the default is 298 ``-I${SRC_libfoo} -I${OBJ_libfoo}`` 299 300``LDADD_libfoo`` 301 This only applies to libs reference via ``DPLIBS``. 302 The default is ``-lfoo``, ``LDADD_*`` provides a hook to 303 instantiate other linker flags at the appropriate point 304 without losing the benfits of ``DPLIBS``. 305 306prog.mk 307------- 308 309Compiles the specified SRCS and links them and the nominated libraries 310into a program. Prog makefiles usually need to list the libraries 311that need to be linked. We prefer use of ``DPLIBS`` but the more 312traditional ``DPADD`` and ``LDADD`` work just as well. 313That is:: 314 315 DPLIBS += ${LIBCRYPTO} 316 317is equivalent to:: 318 319 DPADD += ${LIBCRYPTO} 320 LDADD += -lcrypto 321 322obj.mk 323------ 324 325One of the cool aspects of BSD make, is its support for separating 326object files from the src tree. This is also the source of much 327confusion for people unfamiliar with it. 328 329Traditionally one had to do a separate ``make obj`` pass through the 330tree. If ``MK_AUTO_OBJ`` is set we include auto.obj.mk_. 331 332In fact if ``MKOBJDIRS`` is set to "auto", `sys.mk`_ will set 333``MK_AUTO_OBJ=yes`` and include auto.obj.mk_ since it is best done early. 334 335auto.obj.mk 336----------- 337 338Creates object dirs and leverages the ``.OBJDIR`` target introduced 339some years ago to NetBSD make, to use them. 340 341Note that if ``auto.obj.mk`` is to be used it should be included 342early - before bmake has established ``.PATH``, thus we include it 343from ``sys.mk`` rather than ``obj.mk``. 344 345subdir.mk 346--------- 347 348This is the traditional means of walking the tree. A makefile sets 349``SUBDIR`` to the list of sub-dirs to visit. 350 351If ``SUBDIR_MUST_EXIST`` is set, missing directories cause an error, 352otherwise a warning is issued. If you don't even want the warning, 353set ``MISSING_DIR=continue``. 354 355Traditionally, ``subdir.mk`` prints clues as it visits each subdir:: 356 357 ===> ssl 358 ===> ssl/lib 359 ===> ssl/lib/sslfd 360 361you can suppress that - or enhance it by setting ``ECHO_DIR``:: 362 363 # suppress subdir noise 364 ECHO_DIR=: 365 # print time stamps 366 ECHO_DIR=echo @ `date "+%s [%Y-%m-%d %T] "` 367 368I prefer to use `dirdeps.mk`_ which makes ``subdir.mk`` irrelevant. 369 370links.mk 371-------- 372 373Provides rules for processing lists of ``LINKS`` and ``SYMLINKS``. 374Each is expected to be a list of ``link`` and ``target`` pairs 375(``link`` -> ``target``). 376 377The logic is generally in a ``_*_SCRIPT`` which is referenced in a 378``_*_USE`` (``.USE``) target. 379 380The ``_BUILD_*`` forms are identical, but do not use ``${DESTDIR}`` 381and so are useful for creating symlinks during the build phase. 382For example:: 383 384 SYMLINKS += ${.CURDIR}/${MACHINE_ARCH}/include machine 385 header_links: _BUILD_SYMLINKS_USE 386 387 md.o: header_links 388 389would create a symlink called ``machine`` in ``${.OBJDIR}`` pointing to 390``${.CURDIR}/${MACHINE_ARCH}/include`` before compiling ``md.o`` 391 392 393autoconf.mk 394----------- 395 396Deals with running (or generating) GNU autoconf ``configure`` scripts. 397 398dep.mk 399------ 400 401Deals with collecting dependencies. Another useful feature of BSD 402make is the separation of this sort of information into a ``.depend`` 403file. ``MKDEP_CMD`` needs to point to a suitable tool (like mkdeps.sh_) 404 405If ``MK_AUTODEP`` is "yes" it sets ``MKDEP_MK`` to autodep.mk_ by default. 406 407``MKDEP_MK`` can also be set to `auto.dep.mk`_ which is more efficient 408but does not support an explicit ``depend`` target. 409 410autodep.mk 411---------- 412 413Leverages the ``-MD`` feature of recent GCC to collect dependency 414information as a side effect of compilation. With this GCC puts 415dependency info into a ``.d`` file. 416 417Unfortunately GCC bases the name of the ``.d`` file on the name of the 418input rather than the output file, which causes problems when the same 419source is compiled different ways. The latest GCC supports ``-MF`` to 420name the ``.d`` file and ``-MT`` to control the name to put as the 421dependent. 422 423Recent bmake allows dependencies for the ``.END`` target (run at the 424end if everything was successful), and ``autodep.mk`` uses this to 425post process the ``.d`` files into ``.depend``. 426 427auto.dep.mk 428----------- 429 430A much simpler implementation than autodep.mk_ it uses 431``-MF ${.TARGET:T}.d`` 432to avoid possible conflicts during parallel builds. 433This precludes the use of suffix rules to drive ``make depend``, so 434dep.mk_ handles that if specifically requested. 435 436If ``bmake`` is 20160218 or newer, ``auto.dep.mk`` uses ``.dinclude`` 437to includes the ``*.d`` files directly thus avoiding the need to 438create a ``.depend`` file from them. 439 440own.mk 441------ 442 443Normally included by ``init.mk`` (included by ``lib.mk`` and 444``prog.mk`` etc), sets macros for default ownership etc. 445 446It includes ``${MAKECONF}`` if it is defined and exists. 447 448ldorder.mk 449---------- 450 451Leverages ``bmake`` to compute optimal link order for libraries. 452This works nicely and makes refactoring a breeze - so long as you 453have no (or few) cicular dependencies between libraries. 454 455Consider this experimental. 456 457man.mk 458------ 459 460Deals with man pages. 461 462warnings.mk 463----------- 464 465This provides a means of fine grained control over warnings on a per 466``${MACHINE}`` or even file basis. 467 468A makefile sets ``WARNINGS_SET`` to name a list of warnings 469and individual ``W_*`` macros can be used to tweak them. 470For example:: 471 472 WARNINGS_SET = HIGH 473 W_unused_sparc = -Wno-unused 474 475would add all the warnings in ``${HIGH_WARNINGS}`` to CFLAGS, but 476on sparc, ``-Wno-unused`` would replace ``-Wunused``. 477 478You should never need to edit ``warnings.mk``, it will include 479``warnings-sets.mk`` and/or ``local.warnings.mk`` to pick up 480customizations. 481 482rst2htm.mk 483---------- 484 485Logic to simplify generating HTML (and PDF) documents from ReStructuredText. 486 487cython.mk 488--------- 489 490Logic to build Python C interface modules using Cython_ 491 492.. _Cython: http://www.cython.org/ 493 494cc-wrap.mk 495---------- 496 497This makefile leverages two new features in bmake 20220126 and later. 498 499First is the ablity to set target local variables (GNU make has done 500this for ages). 501 502The second (only intersting if using `meta mode`_) 503allows filtering commands before comparison with previous run to 504decide if a target is out-of-date. 505 506In the past, making use of compiler wrappers like ``ccache``, 507``distcc`` or the newer ``icecc`` could get quite ugly. 508Using ``cc-wrap.mk`` it could not be simpler. 509 510jobs.mk 511------- 512 513This should be included by the top-level makefile. 514If you do:: 515 516 make something-jobs 517 518then ``jobs.mk`` will run:: 519 520 make -j${JOB_MAX} someting > ${JOB_LOGDIR}/something.log 2>&1 521 522this ensures you get a build log and JOB_MAX is assumed to be set 523optimally for the host. 524 525META_MODE 526========= 527 528The 20110505 and later versions of ``mk-files`` include a number of 529makefiles contributed by Juniper Networks, Inc. 530These allow the latest version of bmake_ to run in `meta mode`_ 531see `dirdeps.mk`_ and DIRDEPS_BUILD_ below. 532 533.. _`dirdeps.mk`: /help/sjg/dirdeps.htm 534.. _`meta mode`: bmake-meta-mode.htm 535 536DIRDEPS_BUILD 537============= 538 539When the `meta mode`_ was originally done, there was no distinction 540between META_MODE_ and ``DIRDEPS_BUILD``, but as these were integrated 541into FreeBSD it became clear that META_MODE_ could be useful to many 542developers independently of ``DIRDEPS_BUILD``. 543 544Thus today we distinguish between the two. 545We have the following makefiles which are relevant to 546``DIRDEPS_BUILD`` or META_MODE_:: 547 548 share/mk/auto.obj.mk 549 share/mk/dirdeps-cache-update.mk 550 share/mk/dirdeps-options.mk 551 share/mk/dirdeps-targets.mk 552 share/mk/dirdeps.mk 553 share/mk/gendirdeps.mk 554 share/mk/host-target.mk 555 share/mk/install-new.mk 556 share/mk/meta.autodep.mk 557 share/mk/meta.stage.mk 558 share/mk/meta.sys.mk 559 share/mk/meta2deps.py 560 share/mk/meta2deps.sh 561 share/mk/sys.dependfile.mk 562 share/mk/sys.dirdeps.mk 563 564and the following are typically used for customization. 565See `freebsd-meta-mode`_ and `netbsd-meta-mode`_:: 566 567 share/mk/local.dirdeps-build.mk 568 share/mk/local.dirdeps-missing.mk 569 share/mk/local.dirdeps.mk 570 share/mk/local.meta.sys.mk 571 share/mk/local.sys.dirdeps.env.mk 572 share/mk/local.sys.dirdeps.mk 573 share/mk/local.sys.mk 574 575 576Install 577======= 578 579You can use the content of mk.tar.gz_ without installing at all. 580 581The script ``install-mk`` takes care of copying ``*.mk`` into a 582destination directory, and unless told not to, create ``bsd.*.mk`` links 583for ``lib.mk`` etc. 584 585If you just want to create the ``bsd.*.mk`` links in the directory 586where you unpacked the tar file, you can use:: 587 588 ./mk/install-mk ./mk 589 590------ 591 592.. _bmake: bmake.htm 593.. _NetBSD: http://www.netbsd.org/ 594.. _mkdeps.sh: https://www.crufty.net/ftp/pub/sjg/mkdeps.sh 595.. _mk.tar.gz: https://www.crufty.net/ftp/pub/sjg/mk.tar.gz 596.. _`freebsd-meta-mode`: https://www.crufty.net/sjg/docs/freebsd-meta-mode.htm 597.. _`netbsd-meta-mode`: https://www.crufty.net/sjg/docs/netbsd-meta-mode.htm 598 599:Author: sjg@crufty.net 600:Revision: $Id: mk-files.txt,v 1.25 2023/07/14 23:51:11 sjg Exp $ 601:Copyright: Crufty.NET 602