From: jbandela_at_[hidden]
Date: 2000-09-08 22:22:01
I have updated the Token Iterator and Tokenizer concepts. It is
located in BoostUpdate3.zip in TokenIterator directory
In addition, I have provided an implementation of those concepts
along with several samples. The code has been reorganized. Here are
some highlights of the update.
1. punct_space_tokenizer moved to separate file since there are many
ways to implement it.
2. New template ptr_tokenizer_fun. This is analogous to ptr_fun in
the STL and is used to turn a regular function into a Tokenizer
3. Some of the sample tokenizers that don't need to maintain state
informationn are now functions and use ptr_tokenizer_fun
4. New samples
a) A simple tokenizers that filters out non-alpha characters
b) A simple 4 function (no operator precedence) calculator
c) A replacing tokenizer
Of these, I believe c) is the most interesting. You give it a string
to replace and what to replace it with, and when it is incremented
and dereferenced, it will return the character that would have been
there had a search/replace been done. However, it does it without
modifying the orignal sequence. It is also capable of being used with
input iterators. To do such, because it needs to scan ahead, it has
to maintain a buffer. However, the max size of the buffer is the size
of the string to replace with. Check it out and tell me what you
think.
Thanks to everyone who has taken the time to look at the code and
concepts and provide feedback. Please let me know what you think of
this update including concepts, code, and samples.
John R. Bandela