lexer
Source: quickjs-lexer.c — module exports Lexer, Token (and Location, see location)
A configurable, rule-based tokenizer. Rules are regex-driven and grouped into
named states; the lexer yields Token objects.
Lexer
new Lexer(input[, fileName, mode]) // length 1Input
input is a string, ArrayBuffer or typed array (scanned in memory), or an object that is read in
chunks (a Reader, as in include/stream-utils.h):
input | Read as |
|---|---|
fn(buf, len) | a function returning the bytes written to buf (ArrayBuffer), 0 at the end |
{ read(buf, len) } | the same, as a method |
std.open(...) | a std FILE |
const lex = new Lexer(std.open('big.txt', 'r'), Lexer.LONGEST, 'big.txt');The tokens are the same as for the whole text, however the reads are cut. A window of the stream is kept: the unconsumed bytes, 256 bytes of history and the current line. A match that reaches the end of the window, or no match at all, reads more and tries again; reads are synchronous.
With a Reader input:
Token.lexemeis a copy and stays valid;lexemeof the lexer is only valid until the next scan.sizeis the number of bytes read so far;eofis true once the stream ended and all was consumed.peekc(),getc(),skipChars(),skipUntil(),position =andinputthrowTypeError.back()throwsRangeErrorfor a location that left the window.
Scanning methods
| Method | Args | Description |
|---|---|---|
peek() | 0 | Returns the next token id without consuming. |
peekToken() | 0 | Returns the next Token without consuming. |
next() | 0 | Consumes and returns the next token id. |
nextToken() | 0 | Consumes and returns the next Token. |
lex() | 0 | Performs one lexing step. |
setInput(input, fileName) | 1 | Replaces the input being scanned. |
skipBytes(n) / skipChars(n) | 0 | Advance the cursor by bytes / chars. |
skipToken() | 0 | Discards the next token. |
skipUntil(predicate) | 1 | Skips until a condition holds. |
back() | 0 | Pushes the last token back. |
currentLine() | 0 | Returns the text of the current line. |
tokenClass(id) | 1 | Returns the class/name for a token id. |
Rule & state management
| Method | Args | Description |
|---|---|---|
define(name, pattern) | 2 | Defines a named sub-pattern. |
addRule(name, pattern) | 2 | Adds a token rule. |
getRule(id) | 1 | Looks up a rule. |
pushState(state) | 1 | Enters a lexer state (alias begin). |
popState() | 0 | Leaves the current state (alias end). |
topState() | 0 | Returns the active state. |
Properties
| Property | Kind | Description |
|---|---|---|
size | getter | Input size in bytes. |
charPos | getter/setter | Cursor position in characters. |
bytePos | getter | Cursor position in bytes. |
loc | getter | Current Location. |
eof | getter | Whether the end of input is reached. |
mode | getter/setter | Lexer mode flags. |
seq | getter | Token sequence counter. |
byteLength / charLength | getter | Remaining input length. |
state | getter/setter | Active state. |
states | getter | All defined states. |
stateDepth / stateStack | getter | State-stack inspection. |
input | getter | The input text. |
lexeme | getter | The most recent lexeme. |
fileName | getter/setter | Source file name. |
ruleNames / rules | getter | Defined rule names / rules. |
tokens | getter | Collected tokens. |
[Symbol.iterator] yields tokens, so a Lexer can drive for…of.
Static functions
| Function | Args | Description |
|---|---|---|
escape(str) | 1 | Escapes regex/lexer metacharacters. |
unescape(str) | 1 | Reverses escape. |
toString(value) | 1 | String form helper. |
fromFile(path) | 1 | Creates a lexer reading from a file. |
Token
new Token(...) // length 1| Property | Kind | Description |
|---|---|---|
charLength / byteLength | getter | Lexeme length in chars / bytes. |
charRange / byteRange | getter | [start, end] ranges. |
charPos / bytePos | getter | Start offsets. |
loc | getter/setter | Location of the token. |
id | getter | Numeric token id. |
seq | getter | Sequence number. |
type | getter | Token type/rule name (enumerable). |
lexer | getter | Owning lexer. |
rule | getter | The matched rule. |
lexeme | getter | The matched text (enumerable). |
Methods: toString(), [Symbol.toPrimitive]().
Export list
Generated by utilities/doc-api.js from the lexer dump of utilities/api-surface.sh: 87 members. Edit the text above this block, not the block.
Lexer
class, constructible
- static methods:
escape(),fromFile(),toString(),unescape() - static values:
FIRST = 0,LAST = 1,LONGEST = 2,YIELD_ID = 0,YIELD_OBJ = 1 - methods:
addRule(),back(),begin(),currentLine(),define(),end(),getRule(),lex(),next(),nextToken(),peek(),peekToken(),popState(),pushState(),setInput(),skipBytes(),skipChars(),skipToken(),skipUntil(),tokenClass(),topState(),[Symbol.iterator]() - properties:
byteLength (get),bytePos (get),charLength (get),charPos (get/set),eof (get),fileName (get/set),input (get),lexeme (get),loc (get),mode (get/set),ruleNames (get),rules (get),seq (get),size (get),state (get/set),stateDepth (get),stateStack (get),states (get),tokens (get),[Symbol.toStringTag]
Location
class, constructible
- static methods:
count() - methods:
clone(),copy(),equal(),nextChar(),toString(),[Symbol.toPrimitive]() - properties:
byteOffset (get),charOffset (get),column (get),file (get/set),line (get),[Symbol.toStringTag]
Token
class, constructible
- static values:
EOF = -1 - methods:
toString(),[Symbol.toPrimitive]() - properties:
byteLength (get),bytePos (get),byteRange (get),charLength (get),charPos (get),charRange (get),id (get),lexeme (get),lexer (get),loc (get/set),rule (get),seq (get),type (get),[Symbol.toStringTag]