1# @(#)TOUR 8.1 (Berkeley) 5/31/93 2 3NOTE -- This is the original TOUR paper distributed with ash and 4does not represent the current state of the shell. It is provided anyway 5since it provides helpful information for how the shell is structured, 6but be warned that things have changed -- the current shell is 7still under development. 8 9================================================================ 10 11 A Tour through Ash 12 13 Copyright 1989 by Kenneth Almquist. 14 15 16DIRECTORIES: The subdirectory bltin contains commands which can 17be compiled stand-alone. The rest of the source is in the main 18ash directory. 19 20SOURCE CODE GENERATORS: Files whose names begin with "mk" are 21programs that generate source code. A complete list of these 22programs is: 23 24 program input files generates 25 ------- ----------- --------- 26 mkbuiltins builtins.def builtins.h builtins.c 27 mknodes nodetypes nodes.h nodes.c 28 mksyntax - syntax.h syntax.c 29 mktokens - token.h 30 31There are undoubtedly too many of these. 32 33EXCEPTIONS: Code for dealing with exceptions appears in 34exceptions.c. The C language doesn't include exception handling, 35so I implement it using setjmp and longjmp. The global variable 36exception contains the type of exception. EXERROR is raised by 37calling error or errorwithstatus. EXINT is an interrupt. 38 39INTERRUPTS: In an interactive shell, an interrupt will cause an 40EXINT exception to return to the main command loop. (Exception: 41EXINT is not raised if the user traps interrupts using the trap 42command.) The INTOFF and INTON macros (defined in exception.h) 43provide uninterruptible critical sections. Between the execution 44of INTOFF and the execution of INTON, interrupt signals will be 45held for later delivery. INTOFF and INTON can be nested. 46 47MEMALLOC.C: Memalloc.c defines versions of malloc and realloc 48which call error when there is no memory left. It also defines a 49stack oriented memory allocation scheme. Allocating off a stack 50is probably more efficient than allocation using malloc, but the 51big advantage is that when an exception occurs all we have to do 52to free up the memory in use at the time of the exception is to 53restore the stack pointer. The stack is implemented using a 54linked list of blocks. 55 56STPUTC: If the stack were contiguous, it would be easy to store 57strings on the stack without knowing in advance how long the 58string was going to be: 59 p = stackptr; 60 *p++ = c; /* repeated as many times as needed */ 61 stackptr = p; 62The following three macros (defined in memalloc.h) perform these 63operations, but grow the stack if you run off the end: 64 STARTSTACKSTR(p); 65 STPUTC(c, p); /* repeated as many times as needed */ 66 grabstackstr(p); 67 68We now start a top-down look at the code: 69 70MAIN.C: The main routine performs some initialization, executes 71the user's profile if necessary, and calls cmdloop. Cmdloop 72repeatedly parses and executes commands. 73 74OPTIONS.C: This file contains the option processing code. It is 75called from main to parse the shell arguments when the shell is 76invoked, and it also contains the set builtin. The -i and -m op- 77tions (the latter turns on job control) require changes in signal 78handling. The routines setjobctl (in jobs.c) and setinteractive 79(in trap.c) are called to handle changes to these options. 80 81PARSING: The parser code is all in parser.c. A recursive des- 82cent parser is used. Syntax tables (generated by mksyntax) are 83used to classify characters during lexical analysis. There are 84four tables: one for normal use, one for use when inside single 85quotes and dollar single quotes, one for use when inside double 86quotes and one for use in arithmetic. The tables are machine 87dependent because they are indexed by character variables and 88the range of a char varies from machine to machine. 89 90PARSE OUTPUT: The output of the parser consists of a tree of 91nodes. The various types of nodes are defined in the file node- 92types. 93 94Nodes of type NARG are used to represent both words and the con- 95tents of here documents. An early version of ash kept the con- 96tents of here documents in temporary files, but keeping here do- 97cuments in memory typically results in significantly better per- 98formance. It would have been nice to make it an option to use 99temporary files for here documents, for the benefit of small 100machines, but the code to keep track of when to delete the tem- 101porary files was complex and I never fixed all the bugs in it. 102(AT&T has been maintaining the Bourne shell for more than ten 103years, and to the best of my knowledge they still haven't gotten 104it to handle temporary files correctly in obscure cases.) 105 106The text field of a NARG structure points to the text of the 107word. The text consists of ordinary characters and a number of 108special codes defined in parser.h. The special codes are: 109 110 CTLVAR Parameter expansion 111 CTLENDVAR End of parameter expansion 112 CTLBACKQ Command substitution 113 CTLBACKQ|CTLQUOTE Command substitution inside double quotes 114 CTLARI Arithmetic expansion 115 CTLENDARI End of arithmetic expansion 116 CTLESC Escape next character 117 118A variable substitution contains the following elements: 119 120 CTLVAR type name '=' [ alternative-text CTLENDVAR ] 121 122The type field is a single character specifying the type of sub- 123stitution. The possible types are: 124 125 VSNORMAL $var 126 VSMINUS ${var-text} 127 VSMINUS|VSNUL ${var:-text} 128 VSPLUS ${var+text} 129 VSPLUS|VSNUL ${var:+text} 130 VSQUESTION ${var?text} 131 VSQUESTION|VSNUL ${var:?text} 132 VSASSIGN ${var=text} 133 VSASSIGN|VSNUL ${var:=text} 134 VSTRIMLEFT ${var#text} 135 VSTRIMLEFTMAX ${var##text} 136 VSTRIMRIGHT ${var%text} 137 VSTRIMRIGHTMAX ${var%%text} 138 VSLENGTH ${#var} 139 VSERROR delayed error 140 141In addition, the type field will have the VSQUOTE flag set if the 142variable is enclosed in double quotes and the VSLINENO flag if 143LINENO is being expanded (the parameter name is the decimal line 144number). The parameter's name comes next, terminated by an equals 145sign. If the type is not VSNORMAL (including when it is VSLENGTH), 146then the text field in the substitution follows, terminated by a 147CTLENDVAR byte. 148 149The type VSERROR is used to allow parsing bad substitutions like 150${var[7]} and generate an error when they are expanded. 151 152Commands in back quotes are parsed and stored in a linked list. 153The locations of these commands in the string are indicated by 154CTLBACKQ and CTLBACKQ+CTLQUOTE characters, depending upon whether 155the back quotes were enclosed in double quotes. 156 157Arithmetic expansion starts with CTLARI and ends with CTLENDARI. 158 159The character CTLESC escapes the next character, so that in case 160any of the CTL characters mentioned above appear in the input, 161they can be passed through transparently. CTLESC is also used to 162escape '*', '?', '[', and '!' characters which were quoted by the 163user and thus should not be used for file name generation. 164 165CTLESC characters have proved to be particularly tricky to get 166right. In the case of here documents which are not subject to 167variable and command substitution, the parser doesn't insert any 168CTLESC characters to begin with (so the contents of the text 169field can be written without any processing). Other here docu- 170ments, and words which are not subject to file name generation, 171have the CTLESC characters removed during the variable and command 172substitution phase. Words which are subject to file name 173generation have the CTLESC characters removed as part of the file 174name phase. 175 176EXECUTION: Command execution is handled by the following files: 177 eval.c The top level routines. 178 redir.c Code to handle redirection of input and output. 179 jobs.c Code to handle forking, waiting, and job control. 180 exec.c Code to do path searches and the actual exec sys call. 181 expand.c Code to evaluate arguments. 182 var.c Maintains the variable symbol table. Called from expand.c. 183 184EVAL.C: Evaltree recursively executes a parse tree. The exit 185status is returned in the global variable exitstatus. The alter- 186native entry evalbackcmd is called to evaluate commands in back 187quotes. It saves the result in memory if the command is a buil- 188tin; otherwise it forks off a child to execute the command and 189connects the standard output of the child to a pipe. 190 191JOBS.C: To create a process, you call makejob to return a job 192structure, and then call forkshell (passing the job structure as 193an argument) to create the process. Waitforjob waits for a job 194to complete. These routines take care of process groups if job 195control is defined. 196 197REDIR.C: Ash allows file descriptors to be redirected and then 198restored without forking off a child process. This is accom- 199plished by duplicating the original file descriptors. The redir- 200tab structure records where the file descriptors have been dupli- 201cated to. 202 203EXEC.C: The routine find_command locates a command, and enters 204the command in the hash table if it is not already there. The 205third argument specifies whether it is to print an error message 206if the command is not found. (When a pipeline is set up, 207find_command is called for all the commands in the pipeline be- 208fore any forking is done, so to get the commands into the hash 209table of the parent process. But to make command hashing as 210transparent as possible, we silently ignore errors at that point 211and only print error messages if the command cannot be found 212later.) 213 214The routine shellexec is the interface to the exec system call. 215 216EXPAND.C: As the routine argstr generates words by parameter 217expansion, command substitution and arithmetic expansion, it 218performs word splitting on the result. As each word is output, 219the routine expandmeta performs file name generation (if enabled). 220 221VAR.C: Variables are stored in a hash table. Probably we should 222switch to extensible hashing. The variable name is stored in the 223same string as the value (using the format "name=value") so that 224no string copying is needed to create the environment of a com- 225mand. Variables which the shell references internally are preal- 226located so that the shell can reference the values of these vari- 227ables without doing a lookup. 228 229When a program is run, the code in eval.c sticks any environment 230variables which precede the command (as in "PATH=xxx command") in 231the variable table as the simplest way to strip duplicates, and 232then calls "environment" to get the value of the environment. 233 234BUILTIN COMMANDS: The procedures for handling these are scat- 235tered throughout the code, depending on which location appears 236most appropriate. They can be recognized because their names al- 237ways end in "cmd". The mapping from names to procedures is 238specified in the file builtins.def, which is processed by the 239mkbuiltins command. 240 241A builtin command is invoked with argc and argv set up like a 242normal program. A builtin command is allowed to overwrite its 243arguments. Builtin routines can call nextopt to do option pars- 244ing. This is kind of like getopt, but you don't pass argc and 245argv to it. Builtin routines can also call error. This routine 246normally terminates the shell (or returns to the main command 247loop if the shell is interactive), but when called from a non- 248special builtin command it causes the builtin command to 249terminate with an exit status of 2. 250 251The directory bltins contains commands which can be compiled in- 252dependently but can also be built into the shell for efficiency 253reasons. The header file bltin.h takes care of most of the 254differences between the ash and the stand-alone environment. 255The user should call the main routine "main", and #define main to 256be the name of the routine to use when the program is linked into 257ash. This #define should appear before bltin.h is included; 258bltin.h will #undef main if the program is to be compiled 259stand-alone. A similar approach is used for a few utilities from 260bin and usr.bin. 261 262CD.C: This file defines the cd and pwd builtins. 263 264SIGNALS: Trap.c implements the trap command. The routine set- 265signal figures out what action should be taken when a signal is 266received and invokes the signal system call to set the signal ac- 267tion appropriately. When a signal that a user has set a trap for 268is caught, the routine "onsig" sets a flag. The routine dotrap 269is called at appropriate points to actually handle the signal. 270When an interrupt is caught and no trap has been set for that 271signal, the routine "onint" in error.c is called. 272 273OUTPUT: Ash uses its own output routines. There are three out- 274put structures allocated. "Output" represents the standard out- 275put, "errout" the standard error, and "memout" contains output 276which is to be stored in memory. This last is used when a buil- 277tin command appears in backquotes, to allow its output to be col- 278lected without doing any I/O through the UNIX operating system. 279The variables out1 and out2 normally point to output and errout, 280respectively, but they are set to point to memout when appropri- 281ate inside backquotes. 282 283INPUT: The basic input routine is pgetc, which reads from the 284current input file. There is a stack of input files; the current 285input file is the top file on this stack. The code allows the 286input to come from a string rather than a file. (This is for the 287-c option and the "." and eval builtin commands.) The global 288variable plinno is saved and restored when files are pushed and 289popped from the stack. The parser routines store the number of 290the current line in this variable. 291 292DEBUGGING: If DEBUG is defined in shell.h, then the shell will 293write debugging information to the file $HOME/trace. Most of 294this is done using the TRACE macro, which takes a set of printf 295arguments inside two sets of parenthesis. Example: 296"TRACE(("n=%d0, n))". The double parenthesis are necessary be- 297cause the preprocessor can't handle functions with a variable 298number of arguments. Defining DEBUG also causes the shell to 299generate a core dump if it is sent a quit signal. The tracing 300code is in show.c. 301