From: Robert Ramey (ramey_at_[hidden])
Date: 2004-06-05 23:14:49
Darren Cook wrote:
> I'm using boost::serializer between two programs: the first creates the
> data then serializes it, the second loads that data in and analyzes it.
> When I had just 420 data samples it all worked fine: I create a
> vector<Sample> in memory, serialized it and loaded it in the other side.
> When I moved to 7000 data samples the first program, the creator, sucked
> all the memory from the machine and then some.
7000 samples? That doesn't seem very large in today's environment. I can't
see why creating an archive should consume any significant memory at all.
Nothing significant is constructed. Perhaps is a bug somewhere.
In any case, you do raise in interesting issue which would crop up if
serialization were used in something like data logging.
> So I moved to serializing each data sample as it was created.
> To read it back in I changed my code from this:
> std::ifstream input_file(fname,std::ios::binary);
> boost::archive::binary_iarchive archive(input_file);
> std::vector<Sample> samples;
> archive>>samples;
> input_file.close();
> Into this:
> std::ifstream input_file(fname,std::ios::binary);
> boost::archive::binary_iarchive archive(input_file);
> std::vector<Sample> samples;
> samples.reserve(8000);
> while(!input_file.eof()){
> Sample s;
> archive>>s;
> samples.push_back(s);
> }
> input_file.close();
> But every time it fails with an assert [1]. This seems to be when it has
> reached end of file. I cannot know in advance how many data samples I'll
> write to disk, so it seems I have three options:
> A: Make a special "zero" version of Sample to mark end of file
> B: At end of program 1 write the number of samples to a special file,
> and read that in before starting the above loop, so I know when to stop.
> C: Alter the serializer class to throw exception instead of an assert;
> I can then catch it and carry on.
I think I changed the assert/exception to check before loading rather than
after for the next version. This would permit your code above to work as
you expect. I'll have to double check.
> What is the best way? Is adding a terminator byte to boost::serializer an
> option (i.e. so I don't have to make a terminator version of each data
> structure I want to serialize).
> Incidentally my above loop does a copy of each sample as it adds it to
> the vector. Is the above code going to be common enough to make it worth
> including into boost::serializer as a helper function, which could then
> be optimized with in-place construction or something clever like that?
> (And then it could handle the termination handling itself as well.)
The default implementation of STL collections create a new element on the
stack and copy into the collection. Its as clever as I can make it without
getting tripped up in things like exception safety, objects without default
constructors, etc.
Robert Ramey