1 2NOTE -- This is the original TOUR paper distributed with ash and 3does not represent the current state of the shell. It is provided anyway 4since it provides helpful information for how the shell is structured, 5but be warned that things have changed -- the current shell is 6still under development. 7 8================================================================ 9 10 A Tour through Ash 11 12 Copyright 1989 by Kenneth Almquist. 13 14 15DIRECTORIES: The subdirectory bltin contains commands which can 16be compiled stand-alone. The rest of the source is in the main 17ash directory. 18 19SOURCE CODE GENERATORS: Files whose names begin with "mk" are 20programs that generate source code. A complete list of these 21programs is: 22 23 program input files generates 24 ------- ----------- --------- 25 mkbuiltins builtins.def builtins.h builtins.c 26 mknodes nodetypes nodes.h nodes.c 27 mksyntax - syntax.h syntax.c 28 mktokens - token.h 29 30There are undoubtedly too many of these. 31 32EXCEPTIONS: Code for dealing with exceptions appears in 33exceptions.c. The C language doesn't include exception handling, 34so I implement it using setjmp and longjmp. The global variable 35exception contains the type of exception. EXERROR is raised by 36calling error or errorwithstatus. EXINT is an interrupt. 37 38INTERRUPTS: In an interactive shell, an interrupt will cause an 39EXINT exception to return to the main command loop. (Exception: 40EXINT is not raised if the user traps interrupts using the trap 41command.) The INTOFF and INTON macros (defined in exception.h) 42provide uninterruptible critical sections. Between the execution 43of INTOFF and the execution of INTON, interrupt signals will be 44held for later delivery. INTOFF and INTON can be nested. 45 46MEMALLOC.C: Memalloc.c defines versions of malloc and realloc 47which call error when there is no memory left. It also defines a 48stack oriented memory allocation scheme. Allocating off a stack 49is probably more efficient than allocation using malloc, but the 50big advantage is that when an exception occurs all we have to do 51to free up the memory in use at the time of the exception is to 52restore the stack pointer. The stack is implemented using a 53linked list of blocks. 54 55STPUTC: If the stack were contiguous, it would be easy to store 56strings on the stack without knowing in advance how long the 57string was going to be: 58 p = stackptr; 59 *p++ = c; /* repeated as many times as needed */ 60 stackptr = p; 61The following three macros (defined in memalloc.h) perform these 62operations, but grow the stack if you run off the end: 63 STARTSTACKSTR(p); 64 STPUTC(c, p); /* repeated as many times as needed */ 65 grabstackstr(p); 66 67We now start a top-down look at the code: 68 69MAIN.C: The main routine performs some initialization, executes 70the user's profile if necessary, and calls cmdloop. Cmdloop 71repeatedly parses and executes commands. 72 73OPTIONS.C: This file contains the option processing code. It is 74called from main to parse the shell arguments when the shell is 75invoked, and it also contains the set builtin. The -i and -m op- 76tions (the latter turns on job control) require changes in signal 77handling. The routines setjobctl (in jobs.c) and setinteractive 78(in trap.c) are called to handle changes to these options. 79 80PARSING: The parser code is all in parser.c. A recursive des- 81cent parser is used. Syntax tables (generated by mksyntax) are 82used to classify characters during lexical analysis. There are 83four tables: one for normal use, one for use when inside single 84quotes and dollar single quotes, one for use when inside double 85quotes and one for use in arithmetic. The tables are machine 86dependent because they are indexed by character variables and 87the range of a char varies from machine to machine. 88 89PARSE OUTPUT: The output of the parser consists of a tree of 90nodes. The various types of nodes are defined in the file node- 91types. 92 93Nodes of type NARG are used to represent both words and the con- 94tents of here documents. An early version of ash kept the con- 95tents of here documents in temporary files, but keeping here do- 96cuments in memory typically results in significantly better per- 97formance. It would have been nice to make it an option to use 98temporary files for here documents, for the benefit of small 99machines, but the code to keep track of when to delete the tem- 100porary files was complex and I never fixed all the bugs in it. 101(AT&T has been maintaining the Bourne shell for more than ten 102years, and to the best of my knowledge they still haven't gotten 103it to handle temporary files correctly in obscure cases.) 104 105The text field of a NARG structure points to the text of the 106word. The text consists of ordinary characters and a number of 107special codes defined in parser.h. The special codes are: 108 109 CTLVAR Parameter expansion 110 CTLENDVAR End of parameter expansion 111 CTLBACKQ Command substitution 112 CTLBACKQ|CTLQUOTE Command substitution inside double quotes 113 CTLARI Arithmetic expansion 114 CTLENDARI End of arithmetic expansion 115 CTLESC Escape next character 116 117A variable substitution contains the following elements: 118 119 CTLVAR type name '=' [ alternative-text CTLENDVAR ] 120 121The type field is a single character specifying the type of sub- 122stitution. The possible types are: 123 124 VSNORMAL $var 125 VSMINUS ${var-text} 126 VSMINUS|VSNUL ${var:-text} 127 VSPLUS ${var+text} 128 VSPLUS|VSNUL ${var:+text} 129 VSQUESTION ${var?text} 130 VSQUESTION|VSNUL ${var:?text} 131 VSASSIGN ${var=text} 132 VSASSIGN|VSNUL ${var:=text} 133 VSTRIMLEFT ${var#text} 134 VSTRIMLEFTMAX ${var##text} 135 VSTRIMRIGHT ${var%text} 136 VSTRIMRIGHTMAX ${var%%text} 137 VSLENGTH ${#var} 138 VSERROR delayed error 139 140In addition, the type field will have the VSQUOTE flag set if the 141variable is enclosed in double quotes and the VSLINENO flag if 142LINENO is being expanded (the parameter name is the decimal line 143number). The parameter's name comes next, terminated by an equals 144sign. If the type is not VSNORMAL (including when it is VSLENGTH), 145then the text field in the substitution follows, terminated by a 146CTLENDVAR byte. 147 148The type VSERROR is used to allow parsing bad substitutions like 149${var[7]} and generate an error when they are expanded. 150 151Commands in back quotes are parsed and stored in a linked list. 152The locations of these commands in the string are indicated by 153CTLBACKQ and CTLBACKQ+CTLQUOTE characters, depending upon whether 154the back quotes were enclosed in double quotes. 155 156Arithmetic expansion starts with CTLARI and ends with CTLENDARI. 157 158The character CTLESC escapes the next character, so that in case 159any of the CTL characters mentioned above appear in the input, 160they can be passed through transparently. CTLESC is also used to 161escape '*', '?', '[', and '!' characters which were quoted by the 162user and thus should not be used for file name generation. 163 164CTLESC characters have proved to be particularly tricky to get 165right. In the case of here documents which are not subject to 166variable and command substitution, the parser doesn't insert any 167CTLESC characters to begin with (so the contents of the text 168field can be written without any processing). Other here docu- 169ments, and words which are not subject to file name generation, 170have the CTLESC characters removed during the variable and command 171substitution phase. Words which are subject to file name 172generation have the CTLESC characters removed as part of the file 173name phase. 174 175EXECUTION: Command execution is handled by the following files: 176 eval.c The top level routines. 177 redir.c Code to handle redirection of input and output. 178 jobs.c Code to handle forking, waiting, and job control. 179 exec.c Code to do path searches and the actual exec sys call. 180 expand.c Code to evaluate arguments. 181 var.c Maintains the variable symbol table. Called from expand.c. 182 183EVAL.C: Evaltree recursively executes a parse tree. The exit 184status is returned in the global variable exitstatus. The alter- 185native entry evalbackcmd is called to evaluate commands in back 186quotes. It saves the result in memory if the command is a buil- 187tin; otherwise it forks off a child to execute the command and 188connects the standard output of the child to a pipe. 189 190JOBS.C: To create a process, you call makejob to return a job 191structure, and then call forkshell (passing the job structure as 192an argument) to create the process. Waitforjob waits for a job 193to complete. These routines take care of process groups if job 194control is defined. 195 196REDIR.C: Ash allows file descriptors to be redirected and then 197restored without forking off a child process. This is accom- 198plished by duplicating the original file descriptors. The redir- 199tab structure records where the file descriptors have been dupli- 200cated to. 201 202EXEC.C: The routine find_command locates a command, and enters 203the command in the hash table if it is not already there. The 204third argument specifies whether it is to print an error message 205if the command is not found. (When a pipeline is set up, 206find_command is called for all the commands in the pipeline be- 207fore any forking is done, so to get the commands into the hash 208table of the parent process. But to make command hashing as 209transparent as possible, we silently ignore errors at that point 210and only print error messages if the command cannot be found 211later.) 212 213The routine shellexec is the interface to the exec system call. 214 215EXPAND.C: As the routine argstr generates words by parameter 216expansion, command substitution and arithmetic expansion, it 217performs word splitting on the result. As each word is output, 218the routine expandmeta performs file name generation (if enabled). 219 220VAR.C: Variables are stored in a hash table. Probably we should 221switch to extensible hashing. The variable name is stored in the 222same string as the value (using the format "name=value") so that 223no string copying is needed to create the environment of a com- 224mand. Variables which the shell references internally are preal- 225located so that the shell can reference the values of these vari- 226ables without doing a lookup. 227 228When a program is run, the code in eval.c sticks any environment 229variables which precede the command (as in "PATH=xxx command") in 230the variable table as the simplest way to strip duplicates, and 231then calls "environment" to get the value of the environment. 232 233BUILTIN COMMANDS: The procedures for handling these are scat- 234tered throughout the code, depending on which location appears 235most appropriate. They can be recognized because their names al- 236ways end in "cmd". The mapping from names to procedures is 237specified in the file builtins.def, which is processed by the 238mkbuiltins command. 239 240A builtin command is invoked with argc and argv set up like a 241normal program. A builtin command is allowed to overwrite its 242arguments. Builtin routines can call nextopt to do option pars- 243ing. This is kind of like getopt, but you don't pass argc and 244argv to it. Builtin routines can also call error. This routine 245normally terminates the shell (or returns to the main command 246loop if the shell is interactive), but when called from a non- 247special builtin command it causes the builtin command to 248terminate with an exit status of 2. 249 250The directory bltins contains commands which can be compiled in- 251dependently but can also be built into the shell for efficiency 252reasons. The header file bltin.h takes care of most of the 253differences between the ash and the stand-alone environment. 254The user should call the main routine "main", and #define main to 255be the name of the routine to use when the program is linked into 256ash. This #define should appear before bltin.h is included; 257bltin.h will #undef main if the program is to be compiled 258stand-alone. A similar approach is used for a few utilities from 259bin and usr.bin. 260 261CD.C: This file defines the cd and pwd builtins. 262 263SIGNALS: Trap.c implements the trap command. The routine set- 264signal figures out what action should be taken when a signal is 265received and invokes the signal system call to set the signal ac- 266tion appropriately. When a signal that a user has set a trap for 267is caught, the routine "onsig" sets a flag. The routine dotrap 268is called at appropriate points to actually handle the signal. 269When an interrupt is caught and no trap has been set for that 270signal, the routine "onint" in error.c is called. 271 272OUTPUT: Ash uses its own output routines. There are three out- 273put structures allocated. "Output" represents the standard out- 274put, "errout" the standard error, and "memout" contains output 275which is to be stored in memory. This last is used when a buil- 276tin command appears in backquotes, to allow its output to be col- 277lected without doing any I/O through the UNIX operating system. 278The variables out1 and out2 normally point to output and errout, 279respectively, but they are set to point to memout when appropri- 280ate inside backquotes. 281 282INPUT: The basic input routine is pgetc, which reads from the 283current input file. There is a stack of input files; the current 284input file is the top file on this stack. The code allows the 285input to come from a string rather than a file. (This is for the 286-c option and the "." and eval builtin commands.) The global 287variable plinno is saved and restored when files are pushed and 288popped from the stack. The parser routines store the number of 289the current line in this variable. 290 291DEBUGGING: If DEBUG is defined in shell.h, then the shell will 292write debugging information to the file $HOME/trace. Most of 293this is done using the TRACE macro, which takes a set of printf 294arguments inside two sets of parenthesis. Example: 295"TRACE(("n=%d0, n))". The double parenthesis are necessary be- 296cause the preprocessor can't handle functions with a variable 297number of arguments. Defining DEBUG also causes the shell to 298generate a core dump if it is sent a quit signal. The tracing 299code is in show.c. 300