From: jbandela_at_[hidden]
Date: 2000-09-05 21:01:34
I have updated these concepts. The main change is that the Tokenizer
operator() returns an iterator to the beginning of where in the
sequence the current token corresponds to the sequence. The previous
concept, returned the past the end of the last element in the
sequence that corresponded to the current token. In addition, next is
now a reference that is updated by the functor.
The problem with the old approach was that it could not detect when
the sequence had no tokens, but was non-empty.
Finally, thanks to all of you who have commented. I am seeing there
seems to be some concern about complexity. In the current
implemention (not posted yet, will be posted after the concepts are
nailed down) I have moved all the tokenizers to a different file from
the core token_iterator. The core token_iterator class is now 101
lines including whitespace and comments. In addition, a new template
ptr_tokenizer_fun<Iter,TokType> has been added. It is analogous to
the standard ptr_fun. It allows an ordinary function that has the
right parameters and return value to be turned into a Tokenizer. In
addition, new sample Tokenizers have been implemented. These include
one that only filters out non-alpha characters. Another is a
calculator parsing tokenizer. This is used to implement a simple four
function (non-operator precedence) calculator. In addition, these
tokenizers are both implemented as functions and use the
ptr_tokenizer_fun.
Thanks again for your time.
John Bandela