Subject: [boost] End of AFIO review thoughts (was: Re: [Boost.AFIO] Formal Review)
From: Niall Douglas (s_sourceforge_at_[hidden])
Date: 2015-09-01 21:50:02
On 1 Sep 2015 a 11:41, Joel FALCOU wrote:
> As a concluding note, I *really* want Niall to keep going on making AFIO
> the reference async file I/O library. Just take the needed time to
> reflect on the afformentionned PAI and implementation issues and
> remember that there is no rush to get there.
Thanks for the review Joel.
This final point is an interesting one, and I think it is worth
discussing a bit further, though everyone note I start my holidays
away from Boost on Thursday and you won't see me again here until
GSoC starts up again in January 2016.
Race free filesystem is far more useful to the C++ community than
async file i/o which really does not deliver what people think it
does without completely rearchitecting their application. Race free
filesystem would be a big value add for reliability and
predictability for any code using the file system because it very
substantially reduces the chance for surprise and surface for
security attacks.
As Beman mentioned a long time ago, for race free filesystem we need
to standardise on some method of representing open file handles, and
then somehow figure out how that fits nicely with STL iostreams. This
is a big mess of difficult and hard, because everybody will have an
opinion right from Boost up to WG21, and moreover most people's
opinions will have no technical basis for being better than most
other people's opinions. There are many, technical merit equivalent,
ways to skin that cat, and piloting a compromise suitable to enough
people to be standardisation capable would be a big multi-year
commitment.
Nevertheless, such a library would be of general value to the C++
community, and moreover WG21 is keen on the idea. It would probably
be just as slow as AFIO's file path consuming APIs i.e. a lot slower
than Boost.Filesystem or STL iostreams path operations, and therefore
probably cannot wholly replace the existing solutions (unless POSIX
fixes the missing functionality we have to workaround with inode
comparing loops - I really wish POSIX would take note of Windows more
on file system, Windows has for the most part a far superior
implementation). But I think the knock on consequences on modernising
STL iostreams to better fit today's C++ world would be enormously
valuable, and indeed the source of much disagreement and debate.
If anyone wished to volunteer to design and write such a library and
take it eventually for standardisation, I would be more than happy to
act as an advisor with hard experience from the coal face, but not as
an implementor. You may be surprised to hear that I don't much care
for conflict, and getting that library past review would involve a
whole ton of conflict and I just don't have the energy given my plans
for the next few years. Other fish to fry, as I will explain.
Regarding AFIO, as you all now realise it is not the library you
thought it would be, but my big hope was and is that this perhaps too
early review has given you all a chance to ponder what a next
generation standardised C++ file system and file i/o library ought to
be, and that might bear fruit in any later review of AFIO in months
or years to come. Many of the reviews felt something like what ASIO
provides for seekable async streams makes a lot more sense through
virtue of being much ligher weight - yet, as I countered, if you want
that then ASIO already has that ready to go for you.
I aim for AFIO to bring something very new to the table: a toolkit of
fundamental portable primitives for bare metal programming of file
systems where "bare metal" refers to the direct, unmodified exposure
of portable filing system semantics to the programmer rather than
"bare metal" in terms of least implementation. Where a particular
platform differs from the standardised abstract portable model
presented by AFIO to the programmer, emulation is performed to make
up the shortfall, and hence the apparent complexity and overheads
relative to a least implementation solution.
That comes, unavoidably, with a very different weighting of what
complexities and relative tradeoffs various operations have than you
are used to, and it will wreck you head initially. But I am confident
that after a few months of it getting into your head space, it'll
start to make a lot more sense especially given the relative fixed
and variable costs of operations on the file system relative to one
another, and the next time you see AFIO it'll hopefully all just make
more sense why it does what it does in the way it does it.
*Especially* when you see the standardised C++ transactional
versioned key-value store based on AFIO I have planned, and the
enormous world of new C++ program designs you can do when you can
just *assume* you have always reliable transactional persistent
versioned storage built into the language runtime and which works
perfectly on everything from embedded C++ right up to the largest
mainframes, works during process bootstrap before main() is called,
copes well with power loss and store damage, and has no issue with
concurrent users on heterogeneous platforms making scaling up from
tiny SSDs to large distributed SANs trivial.
Anyone who has ever programmed on Zope
(https://en.wikipedia.org/wiki/Zope) will need no convincing of just
what a paradigm shift a standardised scalable transactional
persistent store in the runtime affords, and this is why I came out
of lurking on boost-dev to develop these libraries after being
adjunct to the community here since 2002 or so. I have had enough
dealing with inherently unreliable data storage in my applications. I
want truly reliable storage supplied as standard in every major
programming language runtime available. That's my end goal, and AFIO
is but a vital step along that path.
Anyway, I'll stop boring you now. I'll be around here until Thursday
morning British Standard Time for any questions etc, then I'll be
deactivating my boost-dev subscriptions until Jan 2016 to take a long
overdue break from Boost library development which has sucked down
something close to 700 hours since February. Private email will
continue to work of course, and I'll be tipping away at no more than
eight hours per week on Boost.Outcome and the AFIO engine rewrite
until they're done. I'll monitor the Fiber mini-review discussion
starting shortly, but I can already give my vote: it's a YES to
acceptance (I have been following Fiber closely to ensure lightweight
futures will work with it. It's a definite yes, and my congrats to
Oliver for doing such a great job).
Thank you *everybody* for the reviews of AFIO. They have been
enormously valuable, and I am tremendously grateful to you all for
taking the time to review my library.
Niall
-- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/