char_delimiters_separator bug

From: Jeremy Siek (jsiek_at_[hidden])
Date: 2001-08-24 17:05:50

It appears that the char_delimiters_separator cuts on white space no
matter what, even if the returnable and non-returnable arguments are
specified. For example:

#include <string>
#include <iostream>
#include <boost/tokenizer.hpp>

int
main()
{
  std::string s = "This is; a test";
  boost::char_delimiters_separator<char> sep(false, "", ";");
  boost::tokenizer<> line_toks(s);
  std::copy(line_toks.begin(), line_toks.end(),
            std::ostream_iterator<std::string>(std::cout, "\n"));
  return 0;
}

the output of this is:

This
is
a
test

whereas what I would expect the output to be is this instead:

This is
a test

Ciao,
Jeremy

----------------------------------------------------------------------
 Jeremy Siek                          http://php.indiana.edu/~jsiek/
 Ph.D. Student, Indiana Univ. B'ton   email: jsiek_at_[hidden]
 C++ Booster (http://www.boost.org)   office phone: [phone]
----------------------------------------------------------------------