A token can be of type:
literal: the value is the character this token represents. This can be a combination of Unicode codepoints, for example for emojis.
_: superscript and subscript commands.
- command: a command such as \sin
}: begin and end group (use for arguments of commands and for grouping)
esc: start of a special command. Followed by commandliteral tokens.
backslash: start of a special command. Followed by commandliteral tokens.
commandliteral: a-zA-Z for special commands (esc sequence, etc...)
placeholder: a placeholder value meant to be replaced by some actual value
space: one or more space characters (including tab, etc...)