xref: /freebsd/bin/sh/TOUR (revision 271c3a9060f2ee55607ebe146523f888e1db2654)
1#	@(#)TOUR	8.1 (Berkeley) 5/31/93
2# $FreeBSD$
3
4NOTE -- This is the original TOUR paper distributed with ash and
5does not represent the current state of the shell.  It is provided anyway
6since it provides helpful information for how the shell is structured,
7but be warned that things have changed -- the current shell is
8still under development.
9
10================================================================
11
12                       A Tour through Ash
13
14               Copyright 1989 by Kenneth Almquist.
15
16
17DIRECTORIES:  The subdirectory bltin contains commands which can
18be compiled stand-alone.  The rest of the source is in the main
19ash directory.
20
21SOURCE CODE GENERATORS:  Files whose names begin with "mk" are
22programs that generate source code.  A complete list of these
23programs is:
24
25        program         input files         generates
26        -------         -----------         ---------
27        mkbuiltins      builtins            builtins.h builtins.c
28        mkinit          *.c                 init.c
29        mknodes         nodetypes           nodes.h nodes.c
30        mksignames          -               signames.h signames.c
31        mksyntax            -               syntax.h syntax.c
32        mktokens            -               token.h
33        bltin/mkexpr    unary_op binary_op  operators.h operators.c
34
35There are undoubtedly too many of these.  Mkinit searches all the
36C source files for entries looking like:
37
38        INIT {
39              x = 1;    /* executed during initialization */
40        }
41
42        RESET {
43              x = 2;    /* executed when the shell does a longjmp
44                           back to the main command loop */
45        }
46
47        SHELLPROC {
48              x = 3;    /* executed when the shell runs a shell procedure */
49        }
50
51It pulls this code out into routines which are when particular
52events occur.  The intent is to improve modularity by isolating
53the information about which modules need to be explicitly
54initialized/reset within the modules themselves.
55
56Mkinit recognizes several constructs for placing declarations in
57the init.c file.
58        INCLUDE "file.h"
59includes a file.  The storage class MKINIT makes a declaration
60available in the init.c file, for example:
61        MKINIT int funcnest;    /* depth of function calls */
62MKINIT alone on a line introduces a structure or union declara-
63tion:
64        MKINIT
65        struct redirtab {
66              short renamed[10];
67        };
68Preprocessor #define statements are copied to init.c without any
69special action to request this.
70
71INDENTATION:  The ash source is indented in multiples of six
72spaces.  The only study that I have heard of on the subject con-
73cluded that the optimal amount to indent is in the range of four
74to six spaces.  I use six spaces since it is not too big a jump
75from the widely used eight spaces.  If you really hate six space
76indentation, use the adjind (source included) program to change
77it to something else.
78
79EXCEPTIONS:  Code for dealing with exceptions appears in
80exceptions.c.  The C language doesn't include exception handling,
81so I implement it using setjmp and longjmp.  The global variable
82exception contains the type of exception.  EXERROR is raised by
83calling error.  EXINT is an interrupt.  EXSHELLPROC is an excep-
84tion which is raised when a shell procedure is invoked.  The pur-
85pose of EXSHELLPROC is to perform the cleanup actions associated
86with other exceptions.  After these cleanup actions, the shell
87can interpret a shell procedure itself without exec'ing a new
88copy of the shell.
89
90INTERRUPTS:  In an interactive shell, an interrupt will cause an
91EXINT exception to return to the main command loop.  (Exception:
92EXINT is not raised if the user traps interrupts using the trap
93command.)  The INTOFF and INTON macros (defined in exception.h)
94provide uninterruptible critical sections.  Between the execution
95of INTOFF and the execution of INTON, interrupt signals will be
96held for later delivery.  INTOFF and INTON can be nested.
97
98MEMALLOC.C:  Memalloc.c defines versions of malloc and realloc
99which call error when there is no memory left.  It also defines a
100stack oriented memory allocation scheme.  Allocating off a stack
101is probably more efficient than allocation using malloc, but the
102big advantage is that when an exception occurs all we have to do
103to free up the memory in use at the time of the exception is to
104restore the stack pointer.  The stack is implemented using a
105linked list of blocks.
106
107STPUTC:  If the stack were contiguous, it would be easy to store
108strings on the stack without knowing in advance how long the
109string was going to be:
110        p = stackptr;
111        *p++ = c;       /* repeated as many times as needed */
112        stackptr = p;
113The following three macros (defined in memalloc.h) perform these
114operations, but grow the stack if you run off the end:
115        STARTSTACKSTR(p);
116        STPUTC(c, p);   /* repeated as many times as needed */
117        grabstackstr(p);
118
119We now start a top-down look at the code:
120
121MAIN.C:  The main routine performs some initialization, executes
122the user's profile if necessary, and calls cmdloop.  Cmdloop
123repeatedly parses and executes commands.
124
125OPTIONS.C:  This file contains the option processing code.  It is
126called from main to parse the shell arguments when the shell is
127invoked, and it also contains the set builtin.  The -i and -j op-
128tions (the latter turns on job control) require changes in signal
129handling.  The routines setjobctl (in jobs.c) and setinteractive
130(in trap.c) are called to handle changes to these options.
131
132PARSING:  The parser code is all in parser.c.  A recursive des-
133cent parser is used.  Syntax tables (generated by mksyntax) are
134used to classify characters during lexical analysis.  There are
135three tables:  one for normal use, one for use when inside single
136quotes, and one for use when inside double quotes.  The tables
137are machine dependent because they are indexed by character vari-
138ables and the range of a char varies from machine to machine.
139
140PARSE OUTPUT:  The output of the parser consists of a tree of
141nodes.  The various types of nodes are defined in the file node-
142types.
143
144Nodes of type NARG are used to represent both words and the con-
145tents of here documents.  An early version of ash kept the con-
146tents of here documents in temporary files, but keeping here do-
147cuments in memory typically results in significantly better per-
148formance.  It would have been nice to make it an option to use
149temporary files for here documents, for the benefit of small
150machines, but the code to keep track of when to delete the tem-
151porary files was complex and I never fixed all the bugs in it.
152(AT&T has been maintaining the Bourne shell for more than ten
153years, and to the best of my knowledge they still haven't gotten
154it to handle temporary files correctly in obscure cases.)
155
156The text field of a NARG structure points to the text of the
157word.  The text consists of ordinary characters and a number of
158special codes defined in parser.h.  The special codes are:
159
160        CTLVAR              Variable substitution
161        CTLENDVAR           End of variable substitution
162        CTLBACKQ            Command substitution
163        CTLBACKQ|CTLQUOTE   Command substitution inside double quotes
164        CTLESC              Escape next character
165
166A variable substitution contains the following elements:
167
168        CTLVAR type name '=' [ alternative-text CTLENDVAR ]
169
170The type field is a single character specifying the type of sub-
171stitution.  The possible types are:
172
173        VSNORMAL            $var
174        VSMINUS             ${var-text}
175        VSMINUS|VSNUL       ${var:-text}
176        VSPLUS              ${var+text}
177        VSPLUS|VSNUL        ${var:+text}
178        VSQUESTION          ${var?text}
179        VSQUESTION|VSNUL    ${var:?text}
180        VSASSIGN            ${var=text}
181        VSASSIGN|VSNUL      ${var:=text}
182
183In addition, the type field will have the VSQUOTE flag set if the
184variable is enclosed in double quotes.  The name of the variable
185comes next, terminated by an equals sign.  If the type is not
186VSNORMAL, then the text field in the substitution follows, ter-
187minated by a CTLENDVAR byte.
188
189Commands in back quotes are parsed and stored in a linked list.
190The locations of these commands in the string are indicated by
191CTLBACKQ and CTLBACKQ+CTLQUOTE characters, depending upon whether
192the back quotes were enclosed in double quotes.
193
194The character CTLESC escapes the next character, so that in case
195any of the CTL characters mentioned above appear in the input,
196they can be passed through transparently.  CTLESC is also used to
197escape '*', '?', '[', and '!' characters which were quoted by the
198user and thus should not be used for file name generation.
199
200CTLESC characters have proved to be particularly tricky to get
201right.  In the case of here documents which are not subject to
202variable and command substitution, the parser doesn't insert any
203CTLESC characters to begin with (so the contents of the text
204field can be written without any processing).  Other here docu-
205ments, and words which are not subject to splitting and file name
206generation, have the CTLESC characters removed during the vari-
207able and command substitution phase.  Words which are subject to
208splitting and file name generation have the CTLESC characters re-
209moved as part of the file name phase.
210
211EXECUTION:  Command execution is handled by the following files:
212        eval.c     The top level routines.
213        redir.c    Code to handle redirection of input and output.
214        jobs.c     Code to handle forking, waiting, and job control.
215        exec.c     Code to do path searches and the actual exec sys call.
216        expand.c   Code to evaluate arguments.
217        var.c      Maintains the variable symbol table.  Called from expand.c.
218
219EVAL.C:  Evaltree recursively executes a parse tree.  The exit
220status is returned in the global variable exitstatus.  The alter-
221native entry evalbackcmd is called to evaluate commands in back
222quotes.  It saves the result in memory if the command is a buil-
223tin; otherwise it forks off a child to execute the command and
224connects the standard output of the child to a pipe.
225
226JOBS.C:  To create a process, you call makejob to return a job
227structure, and then call forkshell (passing the job structure as
228an argument) to create the process.  Waitforjob waits for a job
229to complete.  These routines take care of process groups if job
230control is defined.
231
232REDIR.C:  Ash allows file descriptors to be redirected and then
233restored without forking off a child process.  This is accom-
234plished by duplicating the original file descriptors.  The redir-
235tab structure records where the file descriptors have been dupli-
236cated to.
237
238EXEC.C:  The routine find_command locates a command, and enters
239the command in the hash table if it is not already there.  The
240third argument specifies whether it is to print an error message
241if the command is not found.  (When a pipeline is set up,
242find_command is called for all the commands in the pipeline be-
243fore any forking is done, so to get the commands into the hash
244table of the parent process.  But to make command hashing as
245transparent as possible, we silently ignore errors at that point
246and only print error messages if the command cannot be found
247later.)
248
249The routine shellexec is the interface to the exec system call.
250
251EXPAND.C:  Arguments are processed in three passes.  The first
252(performed by the routine argstr) performs variable and command
253substitution.  The second (ifsbreakup) performs word splitting
254and the third (expandmeta) performs file name generation.  If the
255"/u" directory is simulated, then when "/u/username" is replaced
256by the user's home directory, the flag "didudir" is set.  This
257tells the cd command that it should print out the directory name,
258just as it would if the "/u" directory were implemented using
259symbolic links.
260
261VAR.C:  Variables are stored in a hash table.  Probably we should
262switch to extensible hashing.  The variable name is stored in the
263same string as the value (using the format "name=value") so that
264no string copying is needed to create the environment of a com-
265mand.  Variables which the shell references internally are preal-
266located so that the shell can reference the values of these vari-
267ables without doing a lookup.
268
269When a program is run, the code in eval.c sticks any environment
270variables which precede the command (as in "PATH=xxx command") in
271the variable table as the simplest way to strip duplicates, and
272then calls "environment" to get the value of the environment.
273There are two consequences of this.  First, if an assignment to
274PATH precedes the command, the value of PATH before the assign-
275ment must be remembered and passed to shellexec.  Second, if the
276program turns out to be a shell procedure, the strings from the
277environment variables which preceded the command must be pulled
278out of the table and replaced with strings obtained from malloc,
279since the former will automatically be freed when the stack (see
280the entry on memalloc.c) is emptied.
281
282BUILTIN COMMANDS:  The procedures for handling these are scat-
283tered throughout the code, depending on which location appears
284most appropriate.  They can be recognized because their names al-
285ways end in "cmd".  The mapping from names to procedures is
286specified in the file builtins, which is processed by the mkbuilt-
287ins command.
288
289A builtin command is invoked with argc and argv set up like a
290normal program.  A builtin command is allowed to overwrite its
291arguments.  Builtin routines can call nextopt to do option pars-
292ing.  This is kind of like getopt, but you don't pass argc and
293argv to it.  Builtin routines can also call error.  This routine
294normally terminates the shell (or returns to the main command
295loop if the shell is interactive), but when called from a builtin
296command it causes the builtin command to terminate with an exit
297status of 2.
298
299The directory bltins contains commands which can be compiled in-
300dependently but can also be built into the shell for efficiency
301reasons.  The makefile in this directory compiles these programs
302in the normal fashion (so that they can be run regardless of
303whether the invoker is ash), but also creates a library named
304bltinlib.a which can be linked with ash.  The header file bltin.h
305takes care of most of the differences between the ash and the
306stand-alone environment.  The user should call the main routine
307"main", and #define main to be the name of the routine to use
308when the program is linked into ash.  This #define should appear
309before bltin.h is included; bltin.h will #undef main if the pro-
310gram is to be compiled stand-alone.
311
312CD.C:  This file defines the cd and pwd builtins.  The pwd com-
313mand runs /bin/pwd the first time it is invoked (unless the user
314has already done a cd to an absolute pathname), but then
315remembers the current directory and updates it when the cd com-
316mand is run, so subsequent pwd commands run very fast.  The main
317complication in the cd command is in the docd command, which
318resolves symbolic links into actual names and informs the user
319where the user ended up if he crossed a symbolic link.
320
321SIGNALS:  Trap.c implements the trap command.  The routine set-
322signal figures out what action should be taken when a signal is
323received and invokes the signal system call to set the signal ac-
324tion appropriately.  When a signal that a user has set a trap for
325is caught, the routine "onsig" sets a flag.  The routine dotrap
326is called at appropriate points to actually handle the signal.
327When an interrupt is caught and no trap has been set for that
328signal, the routine "onint" in error.c is called.
329
330OUTPUT:  Ash uses it's own output routines.  There are three out-
331put structures allocated.  "Output" represents the standard out-
332put, "errout" the standard error, and "memout" contains output
333which is to be stored in memory.  This last is used when a buil-
334tin command appears in backquotes, to allow its output to be col-
335lected without doing any I/O through the UNIX operating system.
336The variables out1 and out2 normally point to output and errout,
337respectively, but they are set to point to memout when appropri-
338ate inside backquotes.
339
340INPUT:  The basic input routine is pgetc, which reads from the
341current input file.  There is a stack of input files; the current
342input file is the top file on this stack.  The code allows the
343input to come from a string rather than a file.  (This is for the
344-c option and the "." and eval builtin commands.)  The global
345variable plinno is saved and restored when files are pushed and
346popped from the stack.  The parser routines store the number of
347the current line in this variable.
348
349DEBUGGING:  If DEBUG is defined in shell.h, then the shell will
350write debugging information to the file $HOME/trace.  Most of
351this is done using the TRACE macro, which takes a set of printf
352arguments inside two sets of parenthesis.  Example:
353"TRACE(("n=%d0, n))".  The double parenthesis are necessary be-
354cause the preprocessor can't handle functions with a variable
355number of arguments.  Defining DEBUG also causes the shell to
356generate a core dump if it is sent a quit signal.  The tracing
357code is in show.c.
358