README.md
1Maintaining Autosetup in the SQLite Tree
2========================================================================
3
4This document provides some tips and reminders for the SQLite
5developers regarding using and maintaining the [Autosetup][]-based
6build infrastructure. It is not an [Autosetup][] reference.
7
8**Table of Contents**:
9
10- [Autosetup API Reference](#apiref)
11- [API Tips](#apitips)
12- [Ensuring TCL Compatibility](#tclcompat)
13- [Design Conventions](#conventions)
14 - Symbolic Names of Feature Flags
15 - Do Not Update Global Shared State
16- [Updating Autosetup](#updating)
17 - ***[Patching Autosetup for Project-local changes](#patching)***
18- [Branch-specific Customization](#branch-customization)
19
20
21------------------------------------------------------------------------
22
23<a name="apiref"></a>
24Autosetup API Reference
25========================================================================
26
27The Autosetup API is quite extensive and can be read either in
28the [files in the `autosetup` dir](/dir/autosetup) or using:
29
30>
31```
32$ ./configure --reference | less
33```
34
35That will include any docs from any TCL files in the `./autosetup` dir
36which contain certain (simple) markup defined by autosetup.
37
38This project's own configuration-related TCL code is spread across the
39following files:
40
41- [proj.tcl][]: project-agnostic utility code for autosetup-driven
42 projects. This file is designed to be shared between this project,
43 other projects managed under the SQLite/Hwaci umbrella
44 (e.g. Fossil), and personal projects of SQLite's developers. It is
45 essentially an amalgamation of a decade's worth of autosetup-related
46 utility code.
47- [sqlite-config.tcl][]: utility code which is too project-specific
48 for `proj.tcl`. We split this out of `auto.def` so that it can be
49 used by both `auto.def` and...
50- [auto.def][]: the primary driver for the `./configure` process.
51 When we talk about "the configure script," we're technically
52 referring to this file, though it actually contains very little
53 of the TCL code.
54- [autoconf/auto.def][]: the main driver script for the "autoconf"
55 bundle's configure script. It is essentially a slightly trimmed-down
56 version of the main `auto.def` file. The `autoconf` dir was ported
57 from the Autotools to Autosetup in the 3.49.0 dev cycle but retains
58 the "autoconf" name to minimize downstream disruption.
59
60
61<a name="apitips"></a>
62Autosetup API Tips
63========================================================================
64
65This section briefly covers only APIs which are frequently useful in
66day-to-day maintenance and might not be immediately recognized as such
67from a casual perusal of the relevant TCL files. The complete docs of
68those with `proj-` prefix can be found in [proj.tcl][] and those with
69an `sqlite-` prefix are in [sqlite-config.tcl][]. The others are part
70of Autosetup's core packages and are scattered around [the TCL files
71in ./autosetup](/dir/autosetup).
72
73In (mostly) alphabetical order:
74
75- **`file-isexec filename`**\
76 Should be used in place of `[file executable]`, as it will also
77 check for `${filename}.exe` on Windows platforms. However, on such
78 platforms it also assumes that _any_ existing file is executable.
79
80- **`get-env VAR ?default?`**\
81 Will fetch an "environment variable" from the first of either: (1) a
82 KEY=VALUE passed to the configure script or (2) the system's
83 environment variables. Not to be confused with `getenv`, which only
84 does the latter and is rarely, if ever, useful in this tree.
85 - **`proj-get-env VAR ?default?`**\
86 Works like `get-env` but will, if that function finds no match,
87 look for a file named `./.env-$VAR` and, if found, return its
88 trimmed contents. This can be used, e.g., to set a developer's
89 local preferences for the default `CFLAGS`.\
90 Tip: adding `-O0` to `.env-CFLAGS` reduces rebuild times
91 considerably at the cost of performance in `make devtest` and the
92 like.
93
94- **`proj-fatal msg`**\
95 Emits `$msg` to stderr and exits with non-zero. Its differences from
96 autosetup's `user-error` are purely cosmetic.
97
98- **`proj-if-opt-truthy flag thenScript ?elseScript?`**\
99 Evals `thenScript` if the given `--flag` is truthy, else it
100 evals the optional `elseScript`.
101
102- **`proj-indented-notice ?-error? ?-notice? msg`**\
103 Breaks its `msg` argument into lines, trims them, and emits them
104 with consistent indentation. Exactly how it emits depends on the
105 flags passed to it (or not), as covered in its docs. This will stick
106 out starkly from normal output and is intended to be used only for
107 important notices.
108
109- **`proj-opt-truthy flag`**\
110 Returns 1 if `--flag`'s value is "truthy," i.e. one of (1, on,
111 enabled, yes, true).
112
113- **`proj-opt-was-provided FLAG`**\
114 Returns 1 if `--FLAG` was explicitly provided to configure,
115 else 0. This distinction can be used to determine, e.g., whether
116 `--with-readline` was provided or whether we're searching for
117 readline by default. In the former case, failure to find it should
118 be treated as fatal, where in the latter case it's not.\
119 Unlike most functions which deal with `--flags`, this one does not
120 validate that `$FLAG` is a registered flag so will not fail fatally
121 if `$FLAG` is not registered as an Autosetup option.
122
123- **`proj-val-truthy value`**\
124 Returns 1 if `$value` is "truthy," See `proj-opt-truthy` for the definition
125 of "truthy."
126
127- **`proj-warn msg`**\
128 Emits `$msg` to stderr. Closely-related is autosetup's `user-notice`
129 (described below).
130
131- **`sqlite-add-feature-flag ?-shell? FLAG...`**\
132 Adds the given feature flag to the CFLAGS which are specific to
133 building libsqlite3. It's intended to be passed one or more
134 `-DSQLITE_ENABLE_...`, or similar, flags. If the `-shell` flag is
135 used then it also passes its arguments to
136 `sqlite-add-shell-opt`. This is a no-op if `FLAG` is not provided or
137 is empty.
138
139- **`sqlite-add-shell-opt FLAG...`**\
140 The shell-specific counterpart of `sqlite-add-feature-flag` which
141 only adds the given flag(s) to the CLI-shell-specific CFLAGS.
142
143- **`sqlite-configure BUILD-NAME {script}`**\
144 This is where all configure `--flags` are defined for all known
145 build modes ("canonical" or "autoconf"). After processing all flags,
146 this function runs `$script`, which contains the build-mode-specific
147 configuration bits, and then runs any finalization bits which are
148 common to all build modes. The `auto.def` files are intended to contain
149 exactly two commands:
150 `use sqlite-config; sqlite-configure BUILD-NAME {script}`
151
152- **`user-notice msg`**\
153 Queues `$msg` to be sent to stderr, but does not emit it until
154 either `show-notices` is called or the next time autosetup would
155 output something (it internally calls `show-notices`). This can be
156 used to generate warnings between a "checking for..." message and
157 its resulting "yes/no/whatever" message in such a way as to not
158 spoil the layout of such messages.
159
160
161<a name="tclcompat"></a>
162Ensuring TCL Compatibility
163========================================================================
164
165One of the significant benefits of using Autosetup is that (A) this
166project uses many TCL scripts in the build process and (B) Autosetup
167comes with a TCL interpreter named [JimTCL][].
168
169It is important that any TCL files used by the configure process and
170makefiles remain compatible with both [JimTCL][] and the canonical
171TCL. Though JimTCL has outstanding compatibility with canonical TCL,
172it does have a few corners with incompatibilities, e.g. regular
173expressions. If a script runs in JimTCL without using any
174JimTCL-specific features, then it's a certainty that it will run in
175canonical TCL as well. The opposite, however, is not _always_ the
176case.
177
178When [`./configure`](/file/configure) is run, it goes through a
179bootstrapping process to find a suitable TCL with which to run the
180autosetup framework. The first step involves [finding or building a
181TCL shell](/file/autosetup/autosetup-find-tclsh). That will first
182search for an available `tclsh` (under several common names,
183e.g. `tclsh8.6`) before falling back to compiling the copy of
184`jimsh0.c` included in the source tree. i.e. it will prefer to use a
185system-installed TCL for running the configure script. Once it finds
186(or builds) a TCL shell, it then runs [a sanity test to ensure that
187the shell is suitable](/file/autosetup/autosetup-test-tclsh) before
188using it to run the main autosetup app.
189
190There are two simple ways to ensure that running of the configure
191process uses JimTCL instead of the canonical `tclsh`, and either
192approach provides equally high assurances about configure script
193compatibility across TCL implementations:
194
1951. Build on a system with no `tclsh` installed in the `$PATH`. In that
196 case, the configure process will fall back to building the in-tree
197 copy of JimTCL.
198
1992. Manually build `./jimsh0` in the top of the checkout with:\
200 `cc -o jimsh0 autosetup/jimsh0.c`\
201 With that in place, the configure script will prefer to use that
202 before looking for a system-level `tclsh`. Be aware, though, that
203 `make distclean` will remove that file.
204
205**Note that `./jimsh0` is distinctly different from the `./jimsh`**
206which gets built for code-generation purposes. The latter requires
207non-default build flags to enable features which are
208platform-dependent, most notably to make its `[file normalize]` work.
209This means, for example, that the configure script and its utility
210APIs must not use `[file normalize]`, but autosetup provides a
211TCL-only implementation of `[file-normalize]` (note the dash) for
212portable use in the configure script. Contrariwise, code-generation
213scripts invoked via `make` may use `[file normalize]`, as they'll use
214`./jimsh` or `tclsh` instead of `./jimsh0`.
215
216
217Known TCL Incompatibilities
218------------------------------------------------------------------------
219
220A summary of known incompatibilities in JimTCL
221
222- **CRNL line endings**: prior to 2025-02-05 `fconfigure -translation ...`
223 was a no-op in JimTCL, and it emits CRNL line endings by default on
224 Windows. Since then, it supports `-translation binary`, which is
225 close enough to `-translation lf` for our purposes. When working
226 with files using the `open` command, it is important to use mode
227 `"rb"` or `"wb"`, as appropriate, so that the output does not get
228 CRNL-mangled on Windows.
229
230- **`file copy`** does not support multiple source files. See
231 [](/info/61f18c96183867fe) for a workaround.
232
233- **Regular expressions**:
234
235 - Patterns treat `\nnn` octal values as back-references (which it
236 does not support). Those can be reformulated as demonstrated in
237 [](/info/aeac23359bb681c0).
238
239 - `regsub` does not support the `\y` flag. A workaround is demonstrated
240 in [](/info/c2e5dd791cce3ec4).
241
242
243<a name="conventions"></a>
244Design Conventions
245========================================================================
246
247This section describes the motivations for the most glaring of the
248build's design decisions, in particular how they deviate from
249historical, or even widely-conventional, practices.
250
251Symbolic Names of Feature Flags
252------------------------------------------------------------------------
253
254Historically, the project's makefile has exclusively used
255`UPPER_UNDERSCORE` form for makefile variables. This build, however,
256primarily uses `X.y` format, where `X` is often a category label,
257e.g. `CFLAGS`, and `y` is the specific instance of that category,
258e.g. `CFLAGS.readline`.
259
260When the configure script exports flags for consumption by filtered
261files, e.g. [Makefile.in][] and the generated
262`sqlite_cfg.h`, it does so in the more conventional `X_Y` form because
263those flags get exported as as C `#define`s to `sqlite_cfg.h`, where
264dots are not permitted.
265
266The `X.y` convention is used in the makefiles primarily because the
267person who did the initial port finds that considerably easier on the
268eyes and fingers. In practice, the `X_Y` form of such exports is used
269exactly once in [Makefile.in][], where it's translated from `@X_Y@`
270into into `X.y` form for consumption by [Makefile.in][] and
271[main.mk][]. For example:
272
273>
274```
275LDFLAGS.shobj = @SHOBJ_LDFLAGS@
276LDFLAGS.zlib = @LDFLAGS_ZLIB@
277LDFLAGS.math = @LDFLAGS_MATH@
278```
279
280(That first one is defined by autosetup, and thus applies "LDFLAGS" as
281the suffix rather than the prefix. Which is more legible is a matter
282of taste, for which there is no accounting.)
283
284
285Do Not Update Global Shared State
286------------------------------------------------------------------------
287
288In both the legacy Autotools-driven build and common Autosetup usage,
289feature tests performed by the configure script may amend global flags
290such as `LIBS`, `LDFLAGS`, and `CFLAGS`[^as-cflags]. That's
291appropriate for a makefile which builds a single deliverable, but less
292so for makefiles which produce multiple deliverables. Drawbacks of
293that approach include:
294
295- It's unlikely that every single deliverable will require the same
296 core set of those flags.
297- It can be difficult to determine the origin of any given change to
298 that global state because those changes are hidden behind voodoo
299 performed outside the immediate visibility of the configure script's
300 maintainer.
301- It can force the maintainers of the configure script to place tests
302 in a specific order so that the resulting flags get applied at
303 the correct time and/or in the correct order.\
304 (A real-life example: before the approach described below was taken
305 to collecting build-time flags, the test for `-rpath` had to come
306 _after_ the test for zlib because the results of the `-rpath` test
307 implicitly modified global state which broke the zlib feature
308 test. Because the feature tests no longer (intentionally) modify
309 shared global state, that is not an issue.)
310
311In this build, cases where feature tests modify global state in such a
312way that it may impact later feature tests are either (A) very
313intentionally defined to do so (e.g. the `--with-wasi-sdk` flag has
314invasive side-effects) or (B) are oversights (i.e. bugs).
315
316This tree's [configure script][auto.def], [utility APIs][proj.tcl],
317[Makefile.in][], and [main.mk][] therefore strive to separate the
318results of any given feature test into its own well-defined
319variables. For example:
320
321- The linker flags for zlib are exported from the configure script as
322 `LDFLAGS_ZLIB`, which [Makefile.in][] and [main.mk][] then expose as
323 `LDFLAGS.zlib`.
324- `CFLAGS_READLINE` (a.k.a. `CFLAGS.readline`) contains the `CFLAGS`
325 needed for including `libreadline`, `libedit`, or `linenoise`, and
326 `LDFLAGS_READLINE` (a.k.a. `LDFLAGS.readline`) is its link-time
327 counterpart.
328
329It is then up to the Makefile to apply and order the flags however is
330appropriate.
331
332At the end of the configure script, the global `CFLAGS` _ideally_
333holds only flags which are either relevant to all targets or, failing
334that, will have no unintended side-effects on any targets. That said:
335clients frequently pass custom `CFLAGS` to `./configure` or `make` to
336set library-level feature toggles, e.g. `-DSQLITE_OMIT_FOO`, in which
337case there is no practical way to avoid "polluting" the builds of
338arbitrary makefile targets with those. _C'est la vie._
339
340
341[^as-cflags]: But see this article for a detailed discussion of how
342 autosetup currently deals specifically with CFLAGS:
343 <https://msteveb.github.io/autosetup/articles/handling-cflags/>
344
345
346<a name="updating"></a>
347Updating Autosetup
348========================================================================
349
350Updating autosetup is, more often than not, painless. It requires having
351a checked-out copy of [the autosetup git repository][autosetup-git]:
352
353>
354```
355$ git clone https://github.com/msteveb/autosetup
356$ cd autosetup
357# Or, if it's already checked out:
358$ git pull
359```
360
361Then, from the top-most directory of an SQLite checkout:
362
363>
364```
365$ /path/to/autosetup-checkout/autosetup --install .
366$ fossil status # show the modified files
367```
368
369Unless the upgrade made any incompatible changes (which is exceedingly
370rare), that's all there is to it. After that's done, **apply a patch
371for the change described in the following section**, test the
372configure process, and check it in.
373
374<a name="patching"></a>
375Patching Autosetup for Project-local Changes
376------------------------------------------------------------------------
377
378Autosetup reserves the flag name **`--debug`** for its own purposes,
379and its own special handling of `--enable-...` flags makes `--debug`
380an alias for `--enable-debug`. As this project has a long history of
381using `--enable-debug`, we patch autosetup to use the name
382`--autosetup-debug` in place of `--debug`. That requires (as of this
383writing) four small edits in [](/file/autosetup/autosetup), as
384demonstrated in [check-in 3296c8d3](/info/3296c8d3).
385
386If autosetup is upgraded and this patch is _not_ applied the invoking
387`./configure` will fail loudly because of the declaration of the
388`debug` flag in `auto.def` - duplicated flags are not permitted.
389
390<a name="branch-customization"></a>
391Branch-specific Customization
392========================================================================
393
394Certain vendor-specific branches require slight configure script
395customization. Rather than editing `sqlite-config.tcl` for this,
396which frequently leads to merge conflicts, the following approach
397is recommended:
398
399In the vendor-specific branch, create a file named
400`autosetup/sqlite-custom.tcl`.
401
402That file should contain the following content...
403
404If flag customization is required, add:
405
406>
407```
408proc sqlite-custom-flags {} {
409 # If any existing --flags require different default values
410 # then call:
411 options-defaults {
412 flag-name new-default-value
413 ...
414 }
415 # ^^^ That will replace the default value but will not update
416 # the --help text, which may lead to some confusion:
417 # https://github.com/msteveb/autosetup/issues/77
418
419 return {
420 {*} {
421 new-flag-name => {Help text}
422 ...
423 }
424 }; #see below
425}
426```
427
428That function must return either an empty string or a list in the form
429used internally by `sqlite-config.tcl:sqlite-configure`.
430
431Next, define:
432
433>
434```
435proc sqlite-custom-handle-flags {} {
436 ... do any custom flag handling here ...
437}
438```
439
440That function, if defined, will be called relatively late in the
441configure process, before any filtered files are generated but after
442all other significant processing.
443
444
445[Autosetup]: https://msteveb.github.io/autosetup/
446[auto.def]: /file/auto.def
447[autoconf/auto.def]: /file/autoconf/auto.def
448[autosetup-git]: https://github.com/msteveb/autosetup
449[proj.tcl]: /file/autosetup/proj.tcl
450[sqlite-config.tcl]: /file/autosetup/sqlite-config.tcl
451[Makefile.in]: /file/Makefile.in
452[main.mk]: /file/main.mk
453[JimTCL]: https://jim.tcl.tk
454