Re: Truncating Direct IO files

["Dmitry A. Kazakov" <mailbox@dmitry-kazakov.de>]

On Tue, 21 Apr 2015 01:20:55 -0700 (PDT), tonyg wrote:

> I use your tables package in nearly everything I do!

I ma glad to hear that.

> I've been reading up
> on oop and tagged types and I'm going to bite the bullet and make another
> attempt at using your persistant storage. 

These are different things.

1. There is an implementation of persistent objects that supports
serialization and deserialization. The object may refer to other objects.
The objects are reference counted and when the last reference disappears
the object, if updated, gets serialized.

This persistence layer can be backed by anything. E.g. a database or by a
direct file I/O with some infrastructure sitting on top of it.

2. There is an implementation of a persistent memory pool where you can
allocate and free memory chunks. It supports stream access if you need to
store something larger than single direct I/O element (block). #1 can sit
on top of #2.

3. There is virtual direct I/O layer that supports transactions and makes
it safe against crashes. #2 can sit on top of #3.

4. There is an implementation of various B-trees (maps) including such that
are similar to a relational table with many keys and many values. #4 can
sit on top of #3.

I cannot tell what of this could be useful for you, because I don't know
what are you trying to do.

> Should I stick to normal records rather than use variants for the persistant storage?

See above.

You can store them directly into the persistent memory pool allocating
storage there. You will need some kind of serialization and desrialization
in the form of stream I/O. The persistent memory pool has stream interface.
If you think that serialization is wasting resources, consider portability.
The same storage file should work on a big-endian and little-endian
machine, right? That's is why you should never use Direct I/O with binary
data or stream attributes. The library provides subroutines to serialize
numeric types. Use them to keep stream I/O portable.

You can use the persistent layer if things are complicated. E.g. objects
dependent on each other. Scopes blurred etc. It is possible to have
persistent objects in situ, e.g. a large container of things, graph, tree,
image. Typically very large objects may be too expensive to deserialize.
They may sit in the persistent storage permanently and be accessed via a
proxy object which is serialized and deserialized instead.

Yes, it is complicated, but nobody said it would not be.

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de