From: jbandela_at_[hidden]
Date: 2001-09-09 21:28:42
I was thinking about the following changes to Tokenizer. I would like
some feedback before I implement them.
1. char_delimiters_separator and offset_separator would take an extra
template parameter to specify whether the tokens are assigned using
tok+=*next; OR
tok.assign(cur,next);
(Thanks to Gennadiy E. Rozental for this idea)
2. escaped_list_sepator would allow specifying multiple characters
for escape, separator(c), and quote. This would solve the problem
that Johan Nilsson brought up about empty tokens
3. escaped_list_separator would take another template parameter, to
specify the functor to be used to process escapes. The default would
be tokenizer_default_escape_processor which would have the same
semantics as present. Also available would be a c_escape processor
that would process the C escapes, and a url_escape_processor which
would process url escapes ( ie + for space, and %hh where hh are hex
digits). The last would ease processing of cgi parameters.
Regards,
John R. Bandela