1Design document for the unified scheme data 2=========================================== 3 4How are things connected? 5------------------------- 6 7The unified scheme takes all its data from the `build.info` files seen 8throughout the source tree. These files hold the minimum information 9needed to build end product files from diverse sources. See the 10section on `build.info` files below. 11 12From the information in `build.info` files, `Configure` builds up an 13information database as a hash table called `%unified_info`, which is 14stored in configdata.pm, found at the top of the build tree (which may 15or may not be the same as the source tree). 16 17[`Configurations/common.tmpl`](common.tmpl) uses the data from `%unified_info` to 18generate the rules for building end product files as well as 19intermediary files with the help of a few functions found in the 20build-file templates. See the section on build-file templates further 21down for more information. 22 23build.info files 24---------------- 25 26As mentioned earlier, `build.info` files are meant to hold the minimum 27information needed to build output files, and therefore only (with a 28few possible exceptions [1]) have information about end products (such 29as scripts, library files and programs) and source files (such as C 30files, C header files, assembler files, etc). Intermediate files such 31as object files are rarely directly referred to in `build.info` files (and 32when they are, it's always with the file name extension `.o`), they are 33inferred by `Configure`. By the same rule of minimalism, end product 34file name extensions (such as `.so`, `.a`, `.exe`, etc) are never mentioned 35in `build.info`. Their file name extensions will be inferred by the 36build-file templates, adapted for the platform they are meant for (see 37sections on `%unified_info` and build-file templates further down). 38 39The variables `PROGRAMS`, `LIBS`, `MODULES` and `SCRIPTS` are used to declare 40end products. There are variants for them with `_NO_INST` as suffix 41(`PROGRAM_NO_INST` etc) to specify end products that shouldn't get installed. 42 43The variables `SOURCE`, `DEPEND`, `INCLUDE` and `DEFINE` are indexed by a 44produced file, and their values are the source used to produce that 45particular produced file, extra dependencies, include directories 46needed, or C macros to be defined. 47 48All their values in all the `build.info` throughout the source tree are 49collected together and form a set of programs, libraries, modules and 50scripts to be produced, source files, dependencies, etc etc etc. 51 52Let's have a pretend example, a very limited contraption of OpenSSL, 53composed of the program `apps/openssl`, the libraries `libssl` and 54`libcrypto`, an module `engines/ossltest` and their sources and 55dependencies. 56 57 # build.info 58 LIBS=libcrypto libssl 59 INCLUDE[libcrypto]=include 60 INCLUDE[libssl]=include 61 DEPEND[libssl]=libcrypto 62 63This is the top directory `build.info` file, and it tells us that two 64libraries are to be built, the include directory `include/` shall be 65used throughout when building anything that will end up in each 66library, and that the library `libssl` depend on the library 67`libcrypto` to function properly. 68 69 # apps/build.info 70 PROGRAMS=openssl 71 SOURCE[openssl]=openssl.c 72 INCLUDE[openssl]=.. ../include 73 DEPEND[openssl]=../libssl 74 75This is the `build.info` file in `apps/`, one may notice that all file 76paths mentioned are relative to the directory the `build.info` file is 77located in. This one tells us that there's a program to be built 78called `apps/openss` (the file name extension will depend on the 79platform and is therefore not mentioned in the `build.info` file). It's 80built from one source file, `apps/openssl.c`, and building it requires 81the use of `.` and `include/` include directories (both are declared 82from the point of view of the `apps/` directory), and that the program 83depends on the library `libssl` to function properly. 84 85 # crypto/build.info 86 LIBS=../libcrypto 87 SOURCE[../libcrypto]=aes.c evp.c cversion.c 88 DEPEND[cversion.o]=buildinf.h 89 90 GENERATE[buildinf.h]=../util/mkbuildinf.pl "$(CC) $(CFLAGS)" "$(PLATFORM)" 91 DEPEND[buildinf.h]=../Makefile 92 DEPEND[../util/mkbuildinf.pl]=../util/Foo.pm 93 94This is the `build.info` file in `crypto/`, and it tells us a little more 95about what's needed to produce `libcrypto`. LIBS is used again to 96declare that `libcrypto` is to be produced. This declaration is 97really unnecessary as it's already mentioned in the top `build.info` 98file, but can make the info file easier to understand. This is to 99show that duplicate information isn't an issue. 100 101This `build.info` file informs us that `libcrypto` is built from a few 102source files, `crypto/aes.c`, `crypto/evp.c` and `crypto/cversion.c`. 103It also shows us that building the object file inferred from 104`crypto/cversion.c` depends on `crypto/buildinf.h`. Finally, it 105also shows the possibility to declare how some files are generated 106using some script, in this case a perl script, and how such scripts 107can be declared to depend on other files, in this case a perl module. 108 109Two things are worth an extra note: 110 111`DEPEND[cversion.o]` mentions an object file. DEPEND indexes is the 112only location where it's valid to mention them 113 114 # ssl/build.info 115 LIBS=../libssl 116 SOURCE[../libssl]=tls.c 117 118This is the build.info file in `ssl/`, and it tells us that the 119library `libssl` is built from the source file `ssl/tls.c`. 120 121 # engines/build.info 122 MODULES=dasync 123 SOURCE[dasync]=e_dasync.c 124 DEPEND[dasync]=../libcrypto 125 INCLUDE[dasync]=../include 126 127 MODULES_NO_INST=ossltest 128 SOURCE[ossltest]=e_ossltest.c 129 DEPEND[ossltest]=../libcrypto.a 130 INCLUDE[ossltest]=../include 131 132This is the `build.info` file in `engines/`, telling us that two modules 133called `engines/dasync` and `engines/ossltest` shall be built, that 134`dasync`'s source is `engines/e_dasync.c` and `ossltest`'s source is 135`engines/e_ossltest.c` and that the include directory `include/` may 136be used when building anything that will be part of these modules. 137Also, both modules depend on the library `libcrypto` to function 138properly. `ossltest` is explicitly linked with the static variant of 139the library `libcrypto`. Finally, only `dasync` is being installed, as 140`ossltest` is only for internal testing. 141 142When `Configure` digests these `build.info` files, the accumulated 143information comes down to this: 144 145 LIBS=libcrypto libssl 146 SOURCE[libcrypto]=crypto/aes.c crypto/evp.c crypto/cversion.c 147 DEPEND[crypto/cversion.o]=crypto/buildinf.h 148 INCLUDE[libcrypto]=include 149 SOURCE[libssl]=ssl/tls.c 150 INCLUDE[libssl]=include 151 DEPEND[libssl]=libcrypto 152 153 PROGRAMS=apps/openssl 154 SOURCE[apps/openssl]=apps/openssl.c 155 INCLUDE[apps/openssl]=. include 156 DEPEND[apps/openssl]=libssl 157 158 MODULES=engines/dasync 159 SOURCE[engines/dasync]=engines/e_dasync.c 160 DEPEND[engines/dasync]=libcrypto 161 INCLUDE[engines/dasync]=include 162 163 MODULES_NO_INST=engines/ossltest 164 SOURCE[engines/ossltest]=engines/e_ossltest.c 165 DEPEND[engines/ossltest]=libcrypto.a 166 INCLUDE[engines/ossltest]=include 167 168 GENERATE[crypto/buildinf.h]=util/mkbuildinf.pl "$(CC) $(CFLAGS)" "$(PLATFORM)" 169 DEPEND[crypto/buildinf.h]=Makefile 170 DEPEND[util/mkbuildinf.pl]=util/Foo.pm 171 172A few notes worth mentioning: 173 174`LIBS` may be used to declare routine libraries only. 175 176`PROGRAMS` may be used to declare programs only. 177 178`MODULES` may be used to declare modules only. 179 180The indexes for `SOURCE` must only be end product files, such as 181libraries, programs or modules. The values of `SOURCE` variables must 182only be source files (possibly generated). 183 184`INCLUDE` and `DEPEND` shows a relationship between different files 185(usually produced files) or between files and directories, such as a 186program depending on a library, or between an object file and some 187extra source file. 188 189When `Configure` processes the `build.info` files, it will take it as 190truth without question, and will therefore perform very few checks. 191If the build tree is separate from the source tree, it will assume 192that all built files and up in the build directory and that all source 193files are to be found in the source tree, if they can be found there. 194`Configure` will assume that source files that can't be found in the 195source tree (such as `crypto/bildinf.h` in the example above) are 196generated and will be found in the build tree. 197 198The `%unified_info` database 199---------------------------- 200 201The information in all the `build.info` get digested by `Configure` and 202collected into the `%unified_info` database, divided into the following 203indexes: 204 205 depends => a hash table containing 'file' => [ 'dependency' ... ] 206 pairs. These are directly inferred from the DEPEND 207 variables in build.info files. 208 209 modules => a list of modules. These are directly inferred from 210 the MODULES variable in build.info files. 211 212 generate => a hash table containing 'file' => [ 'generator' ... ] 213 pairs. These are directly inferred from the GENERATE 214 variables in build.info files. 215 216 includes => a hash table containing 'file' => [ 'include' ... ] 217 pairs. These are directly inferred from the INCLUDE 218 variables in build.info files. 219 220 install => a hash table containing 'type' => [ 'file' ... ] pairs. 221 The types are 'programs', 'libraries', 'modules' and 222 'scripts', and the array of files list the files of 223 that type that should be installed. 224 225 libraries => a list of libraries. These are directly inferred from 226 the LIBS variable in build.info files. 227 228 programs => a list of programs. These are directly inferred from 229 the PROGRAMS variable in build.info files. 230 231 scripts => a list of scripts. There are directly inferred from 232 the SCRIPTS variable in build.info files. 233 234 sources => a hash table containing 'file' => [ 'sourcefile' ... ] 235 pairs. These are indirectly inferred from the SOURCE 236 variables in build.info files. Object files are 237 mentioned in this hash table, with source files from 238 SOURCE variables, and AS source files for programs and 239 libraries. 240 241 shared_sources => 242 a hash table just like 'sources', but only as source 243 files (object files) for building shared libraries. 244 245As an example, here is how the `build.info` files example from the 246section above would be digested into a `%unified_info` table: 247 248 our %unified_info = ( 249 "depends" => 250 { 251 "apps/openssl" => 252 [ 253 "libssl", 254 ], 255 "crypto/buildinf.h" => 256 [ 257 "Makefile", 258 ], 259 "crypto/cversion.o" => 260 [ 261 "crypto/buildinf.h", 262 ], 263 "engines/dasync" => 264 [ 265 "libcrypto", 266 ], 267 "engines/ossltest" => 268 [ 269 "libcrypto.a", 270 ], 271 "libssl" => 272 [ 273 "libcrypto", 274 ], 275 "util/mkbuildinf.pl" => 276 [ 277 "util/Foo.pm", 278 ], 279 }, 280 "modules" => 281 [ 282 "engines/dasync", 283 "engines/ossltest", 284 ], 285 "generate" => 286 { 287 "crypto/buildinf.h" => 288 [ 289 "util/mkbuildinf.pl", 290 "\"\$(CC)", 291 "\$(CFLAGS)\"", 292 "\"$(PLATFORM)\"", 293 ], 294 }, 295 "includes" => 296 { 297 "apps/openssl" => 298 [ 299 ".", 300 "include", 301 ], 302 "engines/ossltest" => 303 [ 304 "include" 305 ], 306 "libcrypto" => 307 [ 308 "include", 309 ], 310 "libssl" => 311 [ 312 "include", 313 ], 314 "util/mkbuildinf.pl" => 315 [ 316 "util", 317 ], 318 } 319 "install" => 320 { 321 "modules" => 322 [ 323 "engines/dasync", 324 ], 325 "libraries" => 326 [ 327 "libcrypto", 328 "libssl", 329 ], 330 "programs" => 331 [ 332 "apps/openssl", 333 ], 334 }, 335 "libraries" => 336 [ 337 "libcrypto", 338 "libssl", 339 ], 340 "programs" => 341 [ 342 "apps/openssl", 343 ], 344 "sources" => 345 { 346 "apps/openssl" => 347 [ 348 "apps/openssl.o", 349 ], 350 "apps/openssl.o" => 351 [ 352 "apps/openssl.c", 353 ], 354 "crypto/aes.o" => 355 [ 356 "crypto/aes.c", 357 ], 358 "crypto/cversion.o" => 359 [ 360 "crypto/cversion.c", 361 ], 362 "crypto/evp.o" => 363 [ 364 "crypto/evp.c", 365 ], 366 "engines/e_dasync.o" => 367 [ 368 "engines/e_dasync.c", 369 ], 370 "engines/dasync" => 371 [ 372 "engines/e_dasync.o", 373 ], 374 "engines/e_ossltest.o" => 375 [ 376 "engines/e_ossltest.c", 377 ], 378 "engines/ossltest" => 379 [ 380 "engines/e_ossltest.o", 381 ], 382 "libcrypto" => 383 [ 384 "crypto/aes.c", 385 "crypto/cversion.c", 386 "crypto/evp.c", 387 ], 388 "libssl" => 389 [ 390 "ssl/tls.c", 391 ], 392 "ssl/tls.o" => 393 [ 394 "ssl/tls.c", 395 ], 396 }, 397 ); 398 399As can be seen, everything in `%unified_info` is fairly simple suggest 400of information. Still, it tells us that to build all programs, we 401must build `apps/openssl`, and to build the latter, we will need to 402build all its sources (`apps/openssl.o` in this case) and all the 403other things it depends on (such as `libssl`). All those dependencies 404need to be built as well, using the same logic, so to build `libssl`, 405we need to build `ssl/tls.o` as well as `libcrypto`, and to build the 406latter... 407 408Build-file templates 409-------------------- 410 411Build-file templates are essentially build-files (such as `Makefile` on 412Unix) with perl code fragments mixed in. Those perl code fragment 413will generate all the configuration dependent data, including all the 414rules needed to build end product files and intermediary files alike. 415At a minimum, there must be a perl code fragment that defines a set of 416functions that are used to generates specific build-file rules, to 417build static libraries from object files, to build shared libraries 418from static libraries, to programs from object files and libraries, 419etc. 420 421 generatesrc - function that produces build file lines to generate 422 a source file from some input. 423 424 It's called like this: 425 426 generatesrc(src => "PATH/TO/tobegenerated", 427 generator => [ "generatingfile", ... ] 428 generator_incs => [ "INCL/PATH", ... ] 429 generator_deps => [ "dep1", ... ] 430 incs => [ "INCL/PATH", ... ], 431 deps => [ "dep1", ... ], 432 intent => one of "libs", "dso", "bin" ); 433 434 'src' has the name of the file to be generated. 435 'generator' is the command or part of command to 436 generate the file, of which the first item is 437 expected to be the file to generate from. 438 generatesrc() is expected to analyse and figure out 439 exactly how to apply that file and how to capture 440 the result. 'generator_incs' and 'generator_deps' 441 are include directories and files that the generator 442 file itself depends on. 'incs' and 'deps' are 443 include directories and files that are used if $(CC) 444 is used as an intermediary step when generating the 445 end product (the file indicated by 'src'). 'intent' 446 indicates what the generated file is going to be 447 used for. 448 449 src2obj - function that produces build file lines to build an 450 object file from source files and associated data. 451 452 It's called like this: 453 454 src2obj(obj => "PATH/TO/objectfile", 455 srcs => [ "PATH/TO/sourcefile", ... ], 456 deps => [ "dep1", ... ], 457 incs => [ "INCL/PATH", ... ] 458 intent => one of "lib", "dso", "bin" ); 459 460 'obj' has the intended object file with `.o` 461 extension, src2obj() is expected to change it to 462 something more suitable for the platform. 463 'srcs' has the list of source files to build the 464 object file, with the first item being the source 465 file that directly corresponds to the object file. 466 'deps' is a list of explicit dependencies. 'incs' 467 is a list of include file directories. Finally, 468 'intent' indicates what this object file is going 469 to be used for. 470 471 obj2lib - function that produces build file lines to build a 472 static library file ("libfoo.a" in Unix terms) from 473 object files. 474 475 called like this: 476 477 obj2lib(lib => "PATH/TO/libfile", 478 objs => [ "PATH/TO/objectfile", ... ]); 479 480 'lib' has the intended library file name *without* 481 extension, obj2lib is expected to add that. 'objs' 482 has the list of object files to build this library. 483 484 libobj2shlib - backward compatibility function that's used the 485 same way as obj2shlib (described next), and was 486 expected to build the shared library from the 487 corresponding static library when that was suitable. 488 NOTE: building a shared library from a static 489 library is now DEPRECATED, as they no longer share 490 object files. Attempting to do this will fail. 491 492 obj2shlib - function that produces build file lines to build a 493 shareable object library file ("libfoo.so" in Unix 494 terms) from the corresponding object files. 495 496 called like this: 497 498 obj2shlib(shlib => "PATH/TO/shlibfile", 499 lib => "PATH/TO/libfile", 500 objs => [ "PATH/TO/objectfile", ... ], 501 deps => [ "PATH/TO/otherlibfile", ... ]); 502 503 'lib' has the base (static) library file name 504 *without* extension. This is useful in case 505 supporting files are needed (such as import 506 libraries on Windows). 507 'shlib' has the corresponding shared library name 508 *without* extension. 'deps' has the list of other 509 libraries (also *without* extension) this library 510 needs to be linked with. 'objs' has the list of 511 object files to build this library. 512 513 obj2dso - function that produces build file lines to build a 514 dynamic shared object file from object files. 515 516 called like this: 517 518 obj2dso(lib => "PATH/TO/libfile", 519 objs => [ "PATH/TO/objectfile", ... ], 520 deps => [ "PATH/TO/otherlibfile", 521 ... ]); 522 523 This is almost the same as obj2shlib, but the 524 intent is to build a shareable library that can be 525 loaded in runtime (a "plugin"...). 526 527 obj2bin - function that produces build file lines to build an 528 executable file from object files. 529 530 called like this: 531 532 obj2bin(bin => "PATH/TO/binfile", 533 objs => [ "PATH/TO/objectfile", ... ], 534 deps => [ "PATH/TO/libfile", ... ]); 535 536 'bin' has the intended executable file name 537 *without* extension, obj2bin is expected to add 538 that. 'objs' has the list of object files to build 539 this library. 'deps' has the list of library files 540 (also *without* extension) that the programs needs 541 to be linked with. 542 543 in2script - function that produces build file lines to build a 544 script file from some input. 545 546 called like this: 547 548 in2script(script => "PATH/TO/scriptfile", 549 sources => [ "PATH/TO/infile", ... ]); 550 551 'script' has the intended script file name. 552 'sources' has the list of source files to build the 553 resulting script from. 554 555Along with the build-file templates is the driving template 556[`Configurations/common.tmpl`](common.tmpl), which looks through all the 557information in `%unified_info` and generates all the rulesets to build libraries, 558programs and all intermediate files, using the rule generating 559functions defined in the build-file template. 560 561As an example with the smaller `build.info` set we've seen as an 562example, producing the rules to build `libcrypto` would result in the 563following calls: 564 565 # Note: obj2shlib will only be called if shared libraries are 566 # to be produced. 567 # Note 2: obj2shlib must convert the '.o' extension to whatever 568 # is suitable on the local platform. 569 obj2shlib(shlib => "libcrypto", 570 objs => [ "crypto/aes.o", "crypto/evp.o", "crypto/cversion.o" ], 571 deps => [ ]); 572 573 obj2lib(lib => "libcrypto" 574 objs => [ "crypto/aes.o", "crypto/evp.o", "crypto/cversion.o" ]); 575 576 src2obj(obj => "crypto/aes.o" 577 srcs => [ "crypto/aes.c" ], 578 deps => [ ], 579 incs => [ "include" ], 580 intent => "lib"); 581 582 src2obj(obj => "crypto/evp.o" 583 srcs => [ "crypto/evp.c" ], 584 deps => [ ], 585 incs => [ "include" ], 586 intent => "lib"); 587 588 src2obj(obj => "crypto/cversion.o" 589 srcs => [ "crypto/cversion.c" ], 590 deps => [ "crypto/buildinf.h" ], 591 incs => [ "include" ], 592 intent => "lib"); 593 594 generatesrc(src => "crypto/buildinf.h", 595 generator => [ "util/mkbuildinf.pl", "\"$(CC)", 596 "$(CFLAGS)\"", "\"$(PLATFORM)\"" ], 597 generator_incs => [ "util" ], 598 generator_deps => [ "util/Foo.pm" ], 599 incs => [ ], 600 deps => [ ], 601 intent => "lib"); 602 603The returned strings from all those calls are then concatenated 604together and written to the resulting build-file. 605