From: jbandela_at_[hidden]
Date: 2001-10-19 19:34:55
I have posted a test version of tokenizer to tokenizer_optimize.zip
in the files section. Here are the pertinent changes with respect to
your posts.
1) A single function object is shared across iterators that belong to
a tokenizer
2)Any non-constant state is externalized for a function object
3)Although token_iterator_base still keeps a current iterator in
order to allow all types to be used with tokenizer, it is possible to
partial specialize token_iterator_base for even more efficiency (ie
token_rep where the current iterator is the end iterator of the token)
When I tested token iterator for size, with std::string and STLPort,
with char_delimiters_separator<char>, The size of the iterator was
20, of which 12 was the string.
Let me know what you think,
Regards,
John R. Bandela