summaryrefslogtreecommitdiff
path: root/tokens
AgeCommit message (Collapse)Author
2023-08-04add a crate 'tokens'JSDurand
This crate is responsible for the users to conveniently define their own tokenizers. Right now the user has to tokenize the input string manually, which is not quite what is expected from a parser, from the view point of users.