{ } qjs-modules

lexer

Source: quickjs-lexer.c — module exports Lexer, Token (and Location, see location)

A configurable, rule-based tokenizer. Rules are regex-driven and grouped into named states; the lexer yields Token objects.

Lexer

new Lexer(input[, fileName, mode])   // length 1

Input

input is a string, ArrayBuffer or typed array (scanned in memory), or an object that is read in chunks (a Reader, as in include/stream-utils.h):

inputRead as
fn(buf, len)a function returning the bytes written to buf (ArrayBuffer), 0 at the end
{ read(buf, len) }the same, as a method
std.open(...)a std FILE
const lex = new Lexer(std.open('big.txt', 'r'), Lexer.LONGEST, 'big.txt');

The tokens are the same as for the whole text, however the reads are cut. A window of the stream is kept: the unconsumed bytes, 256 bytes of history and the current line. A match that reaches the end of the window, or no match at all, reads more and tries again; reads are synchronous.

With a Reader input:

  • Token.lexeme is a copy and stays valid; lexeme of the lexer is only valid until the next scan.
  • size is the number of bytes read so far; eof is true once the stream ended and all was consumed.
  • peekc(), getc(), skipChars(), skipUntil(), position = and input throw TypeError.
  • back() throws RangeError for a location that left the window.

Scanning methods

MethodArgsDescription
peek()0Returns the next token id without consuming.
peekToken()0Returns the next Token without consuming.
next()0Consumes and returns the next token id.
nextToken()0Consumes and returns the next Token.
lex()0Performs one lexing step.
setInput(input, fileName)1Replaces the input being scanned.
skipBytes(n) / skipChars(n)0Advance the cursor by bytes / chars.
skipToken()0Discards the next token.
skipUntil(predicate)1Skips until a condition holds.
back()0Pushes the last token back.
currentLine()0Returns the text of the current line.
tokenClass(id)1Returns the class/name for a token id.

Rule & state management

MethodArgsDescription
define(name, pattern)2Defines a named sub-pattern.
addRule(name, pattern)2Adds a token rule.
getRule(id)1Looks up a rule.
pushState(state)1Enters a lexer state (alias begin).
popState()0Leaves the current state (alias end).
topState()0Returns the active state.

Properties

PropertyKindDescription
sizegetterInput size in bytes.
charPosgetter/setterCursor position in characters.
bytePosgetterCursor position in bytes.
locgetterCurrent Location.
eofgetterWhether the end of input is reached.
modegetter/setterLexer mode flags.
seqgetterToken sequence counter.
byteLength / charLengthgetterRemaining input length.
stategetter/setterActive state.
statesgetterAll defined states.
stateDepth / stateStackgetterState-stack inspection.
inputgetterThe input text.
lexemegetterThe most recent lexeme.
fileNamegetter/setterSource file name.
ruleNames / rulesgetterDefined rule names / rules.
tokensgetterCollected tokens.

[Symbol.iterator] yields tokens, so a Lexer can drive for…of.

Static functions

FunctionArgsDescription
escape(str)1Escapes regex/lexer metacharacters.
unescape(str)1Reverses escape.
toString(value)1String form helper.
fromFile(path)1Creates a lexer reading from a file.

Token

new Token(...)   // length 1
PropertyKindDescription
charLength / byteLengthgetterLexeme length in chars / bytes.
charRange / byteRangegetter[start, end] ranges.
charPos / bytePosgetterStart offsets.
locgetter/setterLocation of the token.
idgetterNumeric token id.
seqgetterSequence number.
typegetterToken type/rule name (enumerable).
lexergetterOwning lexer.
rulegetterThe matched rule.
lexemegetterThe matched text (enumerable).

Methods: toString(), [Symbol.toPrimitive]().

Export list

Generated by utilities/doc-api.js from the lexer dump of utilities/api-surface.sh: 87 members. Edit the text above this block, not the block.

Lexer

class, constructible

  • static methods: escape(), fromFile(), toString(), unescape()
  • static values: FIRST = 0, LAST = 1, LONGEST = 2, YIELD_ID = 0, YIELD_OBJ = 1
  • methods: addRule(), back(), begin(), currentLine(), define(), end(), getRule(), lex(), next(), nextToken(), peek(), peekToken(), popState(), pushState(), setInput(), skipBytes(), skipChars(), skipToken(), skipUntil(), tokenClass(), topState(), [Symbol.iterator]()
  • properties: byteLength (get), bytePos (get), charLength (get), charPos (get/set), eof (get), fileName (get/set), input (get), lexeme (get), loc (get), mode (get/set), ruleNames (get), rules (get), seq (get), size (get), state (get/set), stateDepth (get), stateStack (get), states (get), tokens (get), [Symbol.toStringTag]

Location

class, constructible

  • static methods: count()
  • methods: clone(), copy(), equal(), nextChar(), toString(), [Symbol.toPrimitive]()
  • properties: byteOffset (get), charOffset (get), column (get), file (get/set), line (get), [Symbol.toStringTag]

Token

class, constructible

  • static values: EOF = -1
  • methods: toString(), [Symbol.toPrimitive]()
  • properties: byteLength (get), bytePos (get), byteRange (get), charLength (get), charPos (get), charRange (get), id (get), lexeme (get), lexer (get), loc (get/set), rule (get), seq (get), type (get), [Symbol.toStringTag]