Now I tested and discussed with claude, and had a working version of
this project.  The rest is to port the claude version to my own
writings.  So this serves as the starting point before that attempt.

In the new attempt the design is better, and a number of bugs were
found and supposedly fixed, so let us see how this goes at the end.

I think I should make a new sub-crate dedicated to the tokenization
process instead.

Previously the errors emitted while reading abnf grammars reported
incorrect indices.  Now this is fixed.

This makes it easier if I want to debug things.

* grammar/src/label.rs (set_end_option): This function replaces the
  old function `open_end`, as this new function is more general than
  the old one, and there is no specific situation where we only need
  to open the end of a node without the need to close the node in an
  `if` statement.

This bug caused a plain unambiguous grammar to become ambiguous.

Funnily enough, this bug revealed a lot of bugs in the code for
handling forests.  I guess this is an unexpected surprise.  :D

Now the binding part is finished.

What remains is a bug encountered when planting a fragment to the
forest which intersects a packed node, which would lead to invalid
forests.  This will also cause problem when planting a packed
fragment, but until now my testing grammars do not produce packed
fragments, so this problem is not encountered yet.

I am still figuring out efficient ways to solve this problem.

There were two main issues in the previous version.

One is that there are lots of duplications of nodes when manipulating
the forest.  This does not mean that labels repeat: by the use of the
data type this cannot happen.  What happened is that there were cloned
nodes whose children are exactly equal.  In this case there is no need
to clone that node in the first place.  This is now fixed by checking
carefully before cloning, so that we do not clone unnecessary nodes.

The other issue, which is perhaps more important, is that there are
nodes which are not closed.  This means that when there should be a
reuction of grammar rules, the forest does not mark the corresponding
node as already reduced.  The incorrect forests thus caused is hard to
fix: I tried several different approaches to fix it afterwards, but
all to no avail.  I also tried to record enough information to fix
these nodes during the manipulations.  It turned out that recording
nodes is a dead end, as I cannot properly syncronize the information
in the forest and the information in the chain-rule machine.  Any
inconsistencies will result in incorrect operations later on.

The approach I finally adapt is to perform every possible reduction at
each step.  This might lead to some more nodes than what we need.  But
those are technically expected to be there after all, and it is easy
to filter them out, so it is fine, from my point of view at the
moment.

Therefore, what remains is to filter those nodes out and connect it to
the holy Emacs.  :D

Generally speaking the algorithm now works correctly and produces the
right shape of forest for the test ambiguous grammar as well.  It does
not correctly perform the "reductions".  It seems that I deliberately
disabled this part of the functionalities in a previous debugging
tour.

So I have to enable it again and see if it works.