add a crate 'tokens'

This crate is responsible for the users to conveniently define their
own tokenizers.  Right now the user has to tokenize the input string
manually, which is not quite what is expected from a parser, from the
view point of users.

1 files changed, 1 insertions, 1 deletions

diff --git a/src/test.c b/src/test.c
index f9ab43b..c1043c8 100644
--- a/src/test.c
+++ b/src/test.c
@@ -25,7 +25,7 @@ main(int argc, char **argv)
 "\n"
 "item         =/ header [ note ] *1( price )\n"
 "\n"
-"header       =  star \"SP\" title %xA *( \"SP\" / %xA )\n"
+"header       =  *1star \"SP\" title %xA *( \"SP\" / %xA )\n"
 "\n"
 "title        =  1*\"TEXT\"\n"
 "\n"