From: jbandela_at_[hidden]
Date: 2001-08-31 13:22:12
Johan
Updated
http://groups.yahoo.com/group/boost/files/token_functions.hpp
I am now thinking that char_delimiters_separator is not the place to
do empty tokens. Since emtpy tokens almost always involve fields, I
think escaped_list_separator is a better place for it. To support
problems like yours I did 2 things
1. Made sure it returned trailing empty tokens
2. Allow you to specify multiple characters as escape,c,quote
Here is an example
string s = "$GPGSA,A,3,25,04,16,06,18,24,,,,,,,1.86,0.92,1.62*0E,";
typedef escaped_list_separator<char> sep_type;
sep_type sep("","$,*","");
tokenizer<sep_type> t(s,sep);
copy(t.begin(),t.end(),ostream_iterator<string>(cout,"\n"));
Regards
John R. Bandela
--- In boost_at_y..., johan.nilsson_at_e... wrote:
> First of all: yes, now it works in my particular case.
>
> Couple of notes though:
>
> * [when allowing empty tokens] char_delimiters_separator outputs
leading,
> but not trailing empty tokens, e.g.:
>
> tokenizing ",test," outputs ("" "test") instead of ("" "test" ""),
which at
> least I would expected.
>
> * Off the top of my head, it seems like escaped_list_separator
should be
> able to do the same thing (output empty tokens). Which leads to the
question
> on _where_ the policy on outputting empty tokens should be adapted,
or?
>
> // Johan
>