Fork me on GitHub


core/lexer# Token

new Token()private

A token can be of type:

  • literal: the value is the character this token represents. This can be a combination of Unicode codepoints, for example for emojis.
  • ^ and _: superscript and subscript commands.
  • command: a command such as \sin or \text or \alpha
  • { and }: begin and end group (use for arguments of commands and for grouping)
  • #: parameter

  • placeholder: a placeholder value meant to be replaced by some actual value

  • space: one or more space characters (including tab, etc...)

    See: TeX:289

value : string
type : string