Lines Matching full:token

44 .I "token number" ,
45 representing the kind of token read.
46 If there is a value associated with that token, it should be assigned
50 The parser and the lexical analyzer must agree on these token numbers in order for
55 For example, suppose that the token name DIGIT has been defined in the declarations section of the
78 The intent is to return a token number of DIGIT, and a value equal to the numerical value of the
81 the identifier DIGIT will be defined as the token number associated
82 with the token DIGIT.
86 to avoid using any token names in the grammar that are reserved
88 token names
94 The token name
99 As mentioned above, the token numbers may be chosen by Yacc or by the user.
101 The default token number for a literal
103 Other names are assigned token numbers
106 To assign a token number to a token (including literals),
107 the first appearance of the token name or literal
113 This integer is taken to be the token number of the name or literal.
115 It is important that all token numbers be distinct.
117 For historical reasons, the endmarker must have token
119 This token number cannot be redefined by the user; thus, all
120 lexical analyzers should be prepared to return 0 or negative as a token number