From: Gennadiy E. Rozental (rogeeff_at_[hidden])
Date: 2001-08-31 09:11:26
Hi, John
while you are looking into token_function I would like to remind my
proposition to make following change:
Index: token_functions.hpp
===================================================================
RCS file: /cvsroot/boost/boost/boost/token_functions.hpp,v
retrieving revision 1.8
diff -r1.8 token_functions.hpp
278c278
< tok+=*next;
---
> tok.assign( next, 1 );
281,284c281,287
< else
< // append all the non delim characters
< for(;next!=end && !is_nonret(*next) && !
is_ret(*next);++next)
< tok+=*next;
---
> else {
> InputIterator curr = next;
> // append all the non delim characters
> for(;next!=end && !is_nonret(*next) && !
is_ret(*next);++next);
>
> tok.assign( curr, next );
> }
( The same can be done in most cases for offset separator function
and can't be done for escaped_list_separator)
Reason: to allow using string class with const semantic.
What do you think about it?
Gennadiy.
--- In boost_at_y..., jbandela_at_u... wrote:
> http://groups.yahoo.com/group/boost/files/token_functions.hpp
>
>
> Here is a version of token_functions that has a preliminary fix for
> empty tokens for char_delimiters_separator. I have not yet put it
> into the boost cvs because I would like to get more feedback first.
>
> To get empty tokens, use the the following constructor
>
> explicit char_delimiters_separator(const Char * returnable, bool
> return_delims=false)
>
> Here is a quick example
>
> string s = ",how,, are,,you, doing";
> char_delimiters_separator<char> sep(",",false);
> tokenizer<> t(s,sep);
> copy(t.begin(),t.end(),ostream_iterator<string>(cout,"\n"));
>
>
>
> Let me know what you think, both about the implementation, and the
> interface.
>
> Note, when returning empty tokens, all separators are returnable.
>
>
> Regards,
>
> John R. Bandela