1# @(#)TOUR 8.1 (Berkeley) 5/31/93 2# $FreeBSD$ 3 4NOTE -- This is the original TOUR paper distributed with ash and 5does not represent the current state of the shell. It is provided anyway 6since it provides helpful information for how the shell is structured, 7but be warned that things have changed -- the current shell is 8still under development. 9 10================================================================ 11 12 A Tour through Ash 13 14 Copyright 1989 by Kenneth Almquist. 15 16 17DIRECTORIES: The subdirectory bltin contains commands which can 18be compiled stand-alone. The rest of the source is in the main 19ash directory. 20 21SOURCE CODE GENERATORS: Files whose names begin with "mk" are 22programs that generate source code. A complete list of these 23programs is: 24 25 program input files generates 26 ------- ----------- --------- 27 mkbuiltins builtins builtins.h builtins.c 28 mkinit *.c init.c 29 mknodes nodetypes nodes.h nodes.c 30 mksyntax - syntax.h syntax.c 31 mktokens - token.h 32 33There are undoubtedly too many of these. Mkinit searches all the 34C source files for entries looking like: 35 36 INIT { 37 x = 1; /* executed during initialization */ 38 } 39 40 RESET { 41 x = 2; /* executed when the shell does a longjmp 42 back to the main command loop */ 43 } 44 45It pulls this code out into routines which are when particular 46events occur. The intent is to improve modularity by isolating 47the information about which modules need to be explicitly 48initialized/reset within the modules themselves. 49 50Mkinit recognizes several constructs for placing declarations in 51the init.c file. 52 INCLUDE "file.h" 53includes a file. The storage class MKINIT makes a declaration 54available in the init.c file, for example: 55 MKINIT int funcnest; /* depth of function calls */ 56MKINIT alone on a line introduces a structure or union declara- 57tion: 58 MKINIT 59 struct redirtab { 60 short renamed[10]; 61 }; 62Preprocessor #define statements are copied to init.c without any 63special action to request this. 64 65EXCEPTIONS: Code for dealing with exceptions appears in 66exceptions.c. The C language doesn't include exception handling, 67so I implement it using setjmp and longjmp. The global variable 68exception contains the type of exception. EXERROR is raised by 69calling error. EXINT is an interrupt. 70 71INTERRUPTS: In an interactive shell, an interrupt will cause an 72EXINT exception to return to the main command loop. (Exception: 73EXINT is not raised if the user traps interrupts using the trap 74command.) The INTOFF and INTON macros (defined in exception.h) 75provide uninterruptible critical sections. Between the execution 76of INTOFF and the execution of INTON, interrupt signals will be 77held for later delivery. INTOFF and INTON can be nested. 78 79MEMALLOC.C: Memalloc.c defines versions of malloc and realloc 80which call error when there is no memory left. It also defines a 81stack oriented memory allocation scheme. Allocating off a stack 82is probably more efficient than allocation using malloc, but the 83big advantage is that when an exception occurs all we have to do 84to free up the memory in use at the time of the exception is to 85restore the stack pointer. The stack is implemented using a 86linked list of blocks. 87 88STPUTC: If the stack were contiguous, it would be easy to store 89strings on the stack without knowing in advance how long the 90string was going to be: 91 p = stackptr; 92 *p++ = c; /* repeated as many times as needed */ 93 stackptr = p; 94The following three macros (defined in memalloc.h) perform these 95operations, but grow the stack if you run off the end: 96 STARTSTACKSTR(p); 97 STPUTC(c, p); /* repeated as many times as needed */ 98 grabstackstr(p); 99 100We now start a top-down look at the code: 101 102MAIN.C: The main routine performs some initialization, executes 103the user's profile if necessary, and calls cmdloop. Cmdloop 104repeatedly parses and executes commands. 105 106OPTIONS.C: This file contains the option processing code. It is 107called from main to parse the shell arguments when the shell is 108invoked, and it also contains the set builtin. The -i and -m op- 109tions (the latter turns on job control) require changes in signal 110handling. The routines setjobctl (in jobs.c) and setinteractive 111(in trap.c) are called to handle changes to these options. 112 113PARSING: The parser code is all in parser.c. A recursive des- 114cent parser is used. Syntax tables (generated by mksyntax) are 115used to classify characters during lexical analysis. There are 116four tables: one for normal use, one for use when inside single 117quotes and dollar single quotes, one for use when inside double 118quotes and one for use in arithmetic. The tables are machine 119dependent because they are indexed by character variables and 120the range of a char varies from machine to machine. 121 122PARSE OUTPUT: The output of the parser consists of a tree of 123nodes. The various types of nodes are defined in the file node- 124types. 125 126Nodes of type NARG are used to represent both words and the con- 127tents of here documents. An early version of ash kept the con- 128tents of here documents in temporary files, but keeping here do- 129cuments in memory typically results in significantly better per- 130formance. It would have been nice to make it an option to use 131temporary files for here documents, for the benefit of small 132machines, but the code to keep track of when to delete the tem- 133porary files was complex and I never fixed all the bugs in it. 134(AT&T has been maintaining the Bourne shell for more than ten 135years, and to the best of my knowledge they still haven't gotten 136it to handle temporary files correctly in obscure cases.) 137 138The text field of a NARG structure points to the text of the 139word. The text consists of ordinary characters and a number of 140special codes defined in parser.h. The special codes are: 141 142 CTLVAR Variable substitution 143 CTLENDVAR End of variable substitution 144 CTLBACKQ Command substitution 145 CTLBACKQ|CTLQUOTE Command substitution inside double quotes 146 CTLESC Escape next character 147 148A variable substitution contains the following elements: 149 150 CTLVAR type name '=' [ alternative-text CTLENDVAR ] 151 152The type field is a single character specifying the type of sub- 153stitution. The possible types are: 154 155 VSNORMAL $var 156 VSMINUS ${var-text} 157 VSMINUS|VSNUL ${var:-text} 158 VSPLUS ${var+text} 159 VSPLUS|VSNUL ${var:+text} 160 VSQUESTION ${var?text} 161 VSQUESTION|VSNUL ${var:?text} 162 VSASSIGN ${var=text} 163 VSASSIGN|VSNUL ${var:=text} 164 165In addition, the type field will have the VSQUOTE flag set if the 166variable is enclosed in double quotes. The name of the variable 167comes next, terminated by an equals sign. If the type is not 168VSNORMAL, then the text field in the substitution follows, ter- 169minated by a CTLENDVAR byte. 170 171Commands in back quotes are parsed and stored in a linked list. 172The locations of these commands in the string are indicated by 173CTLBACKQ and CTLBACKQ+CTLQUOTE characters, depending upon whether 174the back quotes were enclosed in double quotes. 175 176The character CTLESC escapes the next character, so that in case 177any of the CTL characters mentioned above appear in the input, 178they can be passed through transparently. CTLESC is also used to 179escape '*', '?', '[', and '!' characters which were quoted by the 180user and thus should not be used for file name generation. 181 182CTLESC characters have proved to be particularly tricky to get 183right. In the case of here documents which are not subject to 184variable and command substitution, the parser doesn't insert any 185CTLESC characters to begin with (so the contents of the text 186field can be written without any processing). Other here docu- 187ments, and words which are not subject to splitting and file name 188generation, have the CTLESC characters removed during the vari- 189able and command substitution phase. Words which are subject to 190splitting and file name generation have the CTLESC characters re- 191moved as part of the file name phase. 192 193EXECUTION: Command execution is handled by the following files: 194 eval.c The top level routines. 195 redir.c Code to handle redirection of input and output. 196 jobs.c Code to handle forking, waiting, and job control. 197 exec.c Code to do path searches and the actual exec sys call. 198 expand.c Code to evaluate arguments. 199 var.c Maintains the variable symbol table. Called from expand.c. 200 201EVAL.C: Evaltree recursively executes a parse tree. The exit 202status is returned in the global variable exitstatus. The alter- 203native entry evalbackcmd is called to evaluate commands in back 204quotes. It saves the result in memory if the command is a buil- 205tin; otherwise it forks off a child to execute the command and 206connects the standard output of the child to a pipe. 207 208JOBS.C: To create a process, you call makejob to return a job 209structure, and then call forkshell (passing the job structure as 210an argument) to create the process. Waitforjob waits for a job 211to complete. These routines take care of process groups if job 212control is defined. 213 214REDIR.C: Ash allows file descriptors to be redirected and then 215restored without forking off a child process. This is accom- 216plished by duplicating the original file descriptors. The redir- 217tab structure records where the file descriptors have been dupli- 218cated to. 219 220EXEC.C: The routine find_command locates a command, and enters 221the command in the hash table if it is not already there. The 222third argument specifies whether it is to print an error message 223if the command is not found. (When a pipeline is set up, 224find_command is called for all the commands in the pipeline be- 225fore any forking is done, so to get the commands into the hash 226table of the parent process. But to make command hashing as 227transparent as possible, we silently ignore errors at that point 228and only print error messages if the command cannot be found 229later.) 230 231The routine shellexec is the interface to the exec system call. 232 233EXPAND.C: Arguments are processed in three passes. The first 234(performed by the routine argstr) performs variable and command 235substitution. The second (ifsbreakup) performs word splitting 236and the third (expandmeta) performs file name generation. 237 238VAR.C: Variables are stored in a hash table. Probably we should 239switch to extensible hashing. The variable name is stored in the 240same string as the value (using the format "name=value") so that 241no string copying is needed to create the environment of a com- 242mand. Variables which the shell references internally are preal- 243located so that the shell can reference the values of these vari- 244ables without doing a lookup. 245 246When a program is run, the code in eval.c sticks any environment 247variables which precede the command (as in "PATH=xxx command") in 248the variable table as the simplest way to strip duplicates, and 249then calls "environment" to get the value of the environment. 250 251BUILTIN COMMANDS: The procedures for handling these are scat- 252tered throughout the code, depending on which location appears 253most appropriate. They can be recognized because their names al- 254ways end in "cmd". The mapping from names to procedures is 255specified in the file builtins, which is processed by the mkbuilt- 256ins command. 257 258A builtin command is invoked with argc and argv set up like a 259normal program. A builtin command is allowed to overwrite its 260arguments. Builtin routines can call nextopt to do option pars- 261ing. This is kind of like getopt, but you don't pass argc and 262argv to it. Builtin routines can also call error. This routine 263normally terminates the shell (or returns to the main command 264loop if the shell is interactive), but when called from a builtin 265command it causes the builtin command to terminate with an exit 266status of 2. 267 268The directory bltins contains commands which can be compiled in- 269dependently but can also be built into the shell for efficiency 270reasons. The makefile in this directory compiles these programs 271in the normal fashion (so that they can be run regardless of 272whether the invoker is ash), but also creates a library named 273bltinlib.a which can be linked with ash. The header file bltin.h 274takes care of most of the differences between the ash and the 275stand-alone environment. The user should call the main routine 276"main", and #define main to be the name of the routine to use 277when the program is linked into ash. This #define should appear 278before bltin.h is included; bltin.h will #undef main if the pro- 279gram is to be compiled stand-alone. 280 281CD.C: This file defines the cd and pwd builtins. 282 283SIGNALS: Trap.c implements the trap command. The routine set- 284signal figures out what action should be taken when a signal is 285received and invokes the signal system call to set the signal ac- 286tion appropriately. When a signal that a user has set a trap for 287is caught, the routine "onsig" sets a flag. The routine dotrap 288is called at appropriate points to actually handle the signal. 289When an interrupt is caught and no trap has been set for that 290signal, the routine "onint" in error.c is called. 291 292OUTPUT: Ash uses it's own output routines. There are three out- 293put structures allocated. "Output" represents the standard out- 294put, "errout" the standard error, and "memout" contains output 295which is to be stored in memory. This last is used when a buil- 296tin command appears in backquotes, to allow its output to be col- 297lected without doing any I/O through the UNIX operating system. 298The variables out1 and out2 normally point to output and errout, 299respectively, but they are set to point to memout when appropri- 300ate inside backquotes. 301 302INPUT: The basic input routine is pgetc, which reads from the 303current input file. There is a stack of input files; the current 304input file is the top file on this stack. The code allows the 305input to come from a string rather than a file. (This is for the 306-c option and the "." and eval builtin commands.) The global 307variable plinno is saved and restored when files are pushed and 308popped from the stack. The parser routines store the number of 309the current line in this variable. 310 311DEBUGGING: If DEBUG is defined in shell.h, then the shell will 312write debugging information to the file $HOME/trace. Most of 313this is done using the TRACE macro, which takes a set of printf 314arguments inside two sets of parenthesis. Example: 315"TRACE(("n=%d0, n))". The double parenthesis are necessary be- 316cause the preprocessor can't handle functions with a variable 317number of arguments. Defining DEBUG also causes the shell to 318generate a core dump if it is sent a quit signal. The tracing 319code is in show.c. 320