1# @(#)TOUR 8.1 (Berkeley) 5/31/93 2# $FreeBSD$ 3 4NOTE -- This is the original TOUR paper distributed with ash and 5does not represent the current state of the shell. It is provided anyway 6since it provides helpful information for how the shell is structured, 7but be warned that things have changed -- the current shell is 8still under development. 9 10================================================================ 11 12 A Tour through Ash 13 14 Copyright 1989 by Kenneth Almquist. 15 16 17DIRECTORIES: The subdirectory bltin contains commands which can 18be compiled stand-alone. The rest of the source is in the main 19ash directory. 20 21SOURCE CODE GENERATORS: Files whose names begin with "mk" are 22programs that generate source code. A complete list of these 23programs is: 24 25 program input files generates 26 ------- ----------- --------- 27 mkbuiltins builtins.def builtins.h builtins.c 28 mknodes nodetypes nodes.h nodes.c 29 mksyntax - syntax.h syntax.c 30 mktokens - token.h 31 32There are undoubtedly too many of these. 33 34EXCEPTIONS: Code for dealing with exceptions appears in 35exceptions.c. The C language doesn't include exception handling, 36so I implement it using setjmp and longjmp. The global variable 37exception contains the type of exception. EXERROR is raised by 38calling error. EXINT is an interrupt. 39 40INTERRUPTS: In an interactive shell, an interrupt will cause an 41EXINT exception to return to the main command loop. (Exception: 42EXINT is not raised if the user traps interrupts using the trap 43command.) The INTOFF and INTON macros (defined in exception.h) 44provide uninterruptible critical sections. Between the execution 45of INTOFF and the execution of INTON, interrupt signals will be 46held for later delivery. INTOFF and INTON can be nested. 47 48MEMALLOC.C: Memalloc.c defines versions of malloc and realloc 49which call error when there is no memory left. It also defines a 50stack oriented memory allocation scheme. Allocating off a stack 51is probably more efficient than allocation using malloc, but the 52big advantage is that when an exception occurs all we have to do 53to free up the memory in use at the time of the exception is to 54restore the stack pointer. The stack is implemented using a 55linked list of blocks. 56 57STPUTC: If the stack were contiguous, it would be easy to store 58strings on the stack without knowing in advance how long the 59string was going to be: 60 p = stackptr; 61 *p++ = c; /* repeated as many times as needed */ 62 stackptr = p; 63The following three macros (defined in memalloc.h) perform these 64operations, but grow the stack if you run off the end: 65 STARTSTACKSTR(p); 66 STPUTC(c, p); /* repeated as many times as needed */ 67 grabstackstr(p); 68 69We now start a top-down look at the code: 70 71MAIN.C: The main routine performs some initialization, executes 72the user's profile if necessary, and calls cmdloop. Cmdloop 73repeatedly parses and executes commands. 74 75OPTIONS.C: This file contains the option processing code. It is 76called from main to parse the shell arguments when the shell is 77invoked, and it also contains the set builtin. The -i and -m op- 78tions (the latter turns on job control) require changes in signal 79handling. The routines setjobctl (in jobs.c) and setinteractive 80(in trap.c) are called to handle changes to these options. 81 82PARSING: The parser code is all in parser.c. A recursive des- 83cent parser is used. Syntax tables (generated by mksyntax) are 84used to classify characters during lexical analysis. There are 85four tables: one for normal use, one for use when inside single 86quotes and dollar single quotes, one for use when inside double 87quotes and one for use in arithmetic. The tables are machine 88dependent because they are indexed by character variables and 89the range of a char varies from machine to machine. 90 91PARSE OUTPUT: The output of the parser consists of a tree of 92nodes. The various types of nodes are defined in the file node- 93types. 94 95Nodes of type NARG are used to represent both words and the con- 96tents of here documents. An early version of ash kept the con- 97tents of here documents in temporary files, but keeping here do- 98cuments in memory typically results in significantly better per- 99formance. It would have been nice to make it an option to use 100temporary files for here documents, for the benefit of small 101machines, but the code to keep track of when to delete the tem- 102porary files was complex and I never fixed all the bugs in it. 103(AT&T has been maintaining the Bourne shell for more than ten 104years, and to the best of my knowledge they still haven't gotten 105it to handle temporary files correctly in obscure cases.) 106 107The text field of a NARG structure points to the text of the 108word. The text consists of ordinary characters and a number of 109special codes defined in parser.h. The special codes are: 110 111 CTLVAR Parameter expansion 112 CTLENDVAR End of parameter expansion 113 CTLBACKQ Command substitution 114 CTLBACKQ|CTLQUOTE Command substitution inside double quotes 115 CTLARI Arithmetic expansion 116 CTLENDARI End of arithmetic expansion 117 CTLESC Escape next character 118 119A variable substitution contains the following elements: 120 121 CTLVAR type name '=' [ alternative-text CTLENDVAR ] 122 123The type field is a single character specifying the type of sub- 124stitution. The possible types are: 125 126 VSNORMAL $var 127 VSMINUS ${var-text} 128 VSMINUS|VSNUL ${var:-text} 129 VSPLUS ${var+text} 130 VSPLUS|VSNUL ${var:+text} 131 VSQUESTION ${var?text} 132 VSQUESTION|VSNUL ${var:?text} 133 VSASSIGN ${var=text} 134 VSASSIGN|VSNUL ${var:=text} 135 VSTRIMLEFT ${var#text} 136 VSTRIMLEFTMAX ${var##text} 137 VSTRIMRIGHT ${var%text} 138 VSTRIMRIGHTMAX ${var%%text} 139 VSLENGTH ${#var} 140 VSERROR delayed error 141 142In addition, the type field will have the VSQUOTE flag set if the 143variable is enclosed in double quotes and the VSLINENO flag if 144LINENO is being expanded (the parameter name is the decimal line 145number). The parameter's name comes next, terminated by an equals 146sign. If the type is not VSNORMAL (including when it is VSLENGTH), 147then the text field in the substitution follows, terminated by a 148CTLENDVAR byte. 149 150The type VSERROR is used to allow parsing bad substitutions like 151${var[7]} and generate an error when they are expanded. 152 153Commands in back quotes are parsed and stored in a linked list. 154The locations of these commands in the string are indicated by 155CTLBACKQ and CTLBACKQ+CTLQUOTE characters, depending upon whether 156the back quotes were enclosed in double quotes. 157 158Arithmetic expansion starts with CTLARI and ends with CTLENDARI. 159 160The character CTLESC escapes the next character, so that in case 161any of the CTL characters mentioned above appear in the input, 162they can be passed through transparently. CTLESC is also used to 163escape '*', '?', '[', and '!' characters which were quoted by the 164user and thus should not be used for file name generation. 165 166CTLESC characters have proved to be particularly tricky to get 167right. In the case of here documents which are not subject to 168variable and command substitution, the parser doesn't insert any 169CTLESC characters to begin with (so the contents of the text 170field can be written without any processing). Other here docu- 171ments, and words which are not subject to file name generation, 172have the CTLESC characters removed during the variable and command 173substitution phase. Words which are subject to file name 174generation have the CTLESC characters removed as part of the file 175name phase. 176 177EXECUTION: Command execution is handled by the following files: 178 eval.c The top level routines. 179 redir.c Code to handle redirection of input and output. 180 jobs.c Code to handle forking, waiting, and job control. 181 exec.c Code to do path searches and the actual exec sys call. 182 expand.c Code to evaluate arguments. 183 var.c Maintains the variable symbol table. Called from expand.c. 184 185EVAL.C: Evaltree recursively executes a parse tree. The exit 186status is returned in the global variable exitstatus. The alter- 187native entry evalbackcmd is called to evaluate commands in back 188quotes. It saves the result in memory if the command is a buil- 189tin; otherwise it forks off a child to execute the command and 190connects the standard output of the child to a pipe. 191 192JOBS.C: To create a process, you call makejob to return a job 193structure, and then call forkshell (passing the job structure as 194an argument) to create the process. Waitforjob waits for a job 195to complete. These routines take care of process groups if job 196control is defined. 197 198REDIR.C: Ash allows file descriptors to be redirected and then 199restored without forking off a child process. This is accom- 200plished by duplicating the original file descriptors. The redir- 201tab structure records where the file descriptors have been dupli- 202cated to. 203 204EXEC.C: The routine find_command locates a command, and enters 205the command in the hash table if it is not already there. The 206third argument specifies whether it is to print an error message 207if the command is not found. (When a pipeline is set up, 208find_command is called for all the commands in the pipeline be- 209fore any forking is done, so to get the commands into the hash 210table of the parent process. But to make command hashing as 211transparent as possible, we silently ignore errors at that point 212and only print error messages if the command cannot be found 213later.) 214 215The routine shellexec is the interface to the exec system call. 216 217EXPAND.C: As the routine argstr generates words by parameter 218expansion, command substitution and arithmetic expansion, it 219performs word splitting on the result. As each word is output, 220the routine expandmeta performs file name generation (if enabled). 221 222VAR.C: Variables are stored in a hash table. Probably we should 223switch to extensible hashing. The variable name is stored in the 224same string as the value (using the format "name=value") so that 225no string copying is needed to create the environment of a com- 226mand. Variables which the shell references internally are preal- 227located so that the shell can reference the values of these vari- 228ables without doing a lookup. 229 230When a program is run, the code in eval.c sticks any environment 231variables which precede the command (as in "PATH=xxx command") in 232the variable table as the simplest way to strip duplicates, and 233then calls "environment" to get the value of the environment. 234 235BUILTIN COMMANDS: The procedures for handling these are scat- 236tered throughout the code, depending on which location appears 237most appropriate. They can be recognized because their names al- 238ways end in "cmd". The mapping from names to procedures is 239specified in the file builtins.def, which is processed by the 240mkbuiltins command. 241 242A builtin command is invoked with argc and argv set up like a 243normal program. A builtin command is allowed to overwrite its 244arguments. Builtin routines can call nextopt to do option pars- 245ing. This is kind of like getopt, but you don't pass argc and 246argv to it. Builtin routines can also call error. This routine 247normally terminates the shell (or returns to the main command 248loop if the shell is interactive), but when called from a non- 249special builtin command it causes the builtin command to 250terminate with an exit status of 2. 251 252The directory bltins contains commands which can be compiled in- 253dependently but can also be built into the shell for efficiency 254reasons. The header file bltin.h takes care of most of the 255differences between the ash and the stand-alone environment. 256The user should call the main routine "main", and #define main to 257be the name of the routine to use when the program is linked into 258ash. This #define should appear before bltin.h is included; 259bltin.h will #undef main if the program is to be compiled 260stand-alone. A similar approach is used for a few utilities from 261bin and usr.bin. 262 263CD.C: This file defines the cd and pwd builtins. 264 265SIGNALS: Trap.c implements the trap command. The routine set- 266signal figures out what action should be taken when a signal is 267received and invokes the signal system call to set the signal ac- 268tion appropriately. When a signal that a user has set a trap for 269is caught, the routine "onsig" sets a flag. The routine dotrap 270is called at appropriate points to actually handle the signal. 271When an interrupt is caught and no trap has been set for that 272signal, the routine "onint" in error.c is called. 273 274OUTPUT: Ash uses its own output routines. There are three out- 275put structures allocated. "Output" represents the standard out- 276put, "errout" the standard error, and "memout" contains output 277which is to be stored in memory. This last is used when a buil- 278tin command appears in backquotes, to allow its output to be col- 279lected without doing any I/O through the UNIX operating system. 280The variables out1 and out2 normally point to output and errout, 281respectively, but they are set to point to memout when appropri- 282ate inside backquotes. 283 284INPUT: The basic input routine is pgetc, which reads from the 285current input file. There is a stack of input files; the current 286input file is the top file on this stack. The code allows the 287input to come from a string rather than a file. (This is for the 288-c option and the "." and eval builtin commands.) The global 289variable plinno is saved and restored when files are pushed and 290popped from the stack. The parser routines store the number of 291the current line in this variable. 292 293DEBUGGING: If DEBUG is defined in shell.h, then the shell will 294write debugging information to the file $HOME/trace. Most of 295this is done using the TRACE macro, which takes a set of printf 296arguments inside two sets of parenthesis. Example: 297"TRACE(("n=%d0, n))". The double parenthesis are necessary be- 298cause the preprocessor can't handle functions with a variable 299number of arguments. Defining DEBUG also causes the shell to 300generate a core dump if it is sent a quit signal. The tracing 301code is in show.c. 302