From: jbandela_at_[hidden]
Date: 2001-06-03 19:57:15
>
> token_iterator.hpp
> token_iterator_generator
> make_token_iterator
> (I did not list token_iterator here because that class should
be
> deleted, see below)
>
> tokenizer.hpp
> tokenizer
>
> token_functions.hpp
> cvs_separator
> csv_escape_error (there's a typo in the name of this one)
> offset_separator
> punct_space_separator
>
> Implementation details (namespace detail)
>
> token_iterator.hpp
> tokenizer_policy
> tokenizer_base (should be renamed token_iterator_base)
>
>
So you are basically saying to rename tokenizer_policy.hpp to
tokenizer_iterator.hpp and put tokenizer_policy in detail?
> The token_iterator class should be deleted. You are attempting to
create a
> templated typedef using inheritance, which I'm afraid doesn't
really work.
> There are corner cases where bad things happen. Also, the
> token_iterator_generator already serves the purpose.
Could you give some examples of when this would cause problems? Right
now I see it as a rather nice convenience class, but I am willing to
be enlightened.
> There are a couple additions I'd like to see to the library. We
need a
> tokenizer function that provides the same kind of functionality as
> strtok(). I wrote up a version of this a while back. I'll dust that
code
> off, write docs, and then post it. Also I'd like to see a tokenizer
> function that uses a regular expression (from regex) to do the
spliting.
>
Doesn't punct_space_separator provide all the functionality in
strtok, namely using single characters as delimiters? However, I
think having the extended functionality of your class would be a
bonus. As to the regex tokenizer function, I do not have much
experience with regex and regular expressions, so I probably would
not be the one to write it. If someone could write one, I would be
glad to include it in the functions.
> On the positive side. I must say I really like the idea of
parameterizing
> the token iterator with the tokenizing function. This design is
really
> nice.
Thanks for your comments,
John R. Bandela