Truncating Direct IO files

Truncating Direct IO files

[tonyg]

I am recording records to disk. Sometimes I replace thes records with a smaller set of records.
    I could delete the file each time but would prefer to truncate the file. I can't see a way of doing this, can you?

Re: Truncating Direct IO files

[Dmitry A. Kazakov]

On Tue, 21 Apr 2015 00:30:37 -0700 (PDT), tonyg wrote:

> I am recording records to disk. Sometimes I replace thes records with a
> smaller set of records.

Do you rewrite all file then? In that case you could simply work with a
copy and delete the original, in a sort of double-buffering manner.

> I could delete the file each time but would prefer to truncate the file. I
> can't see a way of doing this, can you?

There seems no way doing that without OS.

In my implementation of a persistent storage pool I simply maintain the
list of free elements and reuse them later.

BTW, I don't overwrite any file elements within single transaction. I don't
know what you are doing, but consider a scenario when the application or
the whole system crashes in the middle of file updating. Would that corrupt
the data?

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Re: Truncating Direct IO files

[tonyg]

I use your tables package in nearly everything I do! I've been reading up on oop and tagged types and I'm going to bite the bullet and make another attempt at using your persistant storage. 

Should I stick to normal records rather than use variants for the persistant storage?

And yes crashes would corrupt my system :)

Re: Truncating Direct IO files

[Dmitry A. Kazakov]

On Tue, 21 Apr 2015 01:20:55 -0700 (PDT), tonyg wrote:

> I use your tables package in nearly everything I do!

I ma glad to hear that.

> I've been reading up
> on oop and tagged types and I'm going to bite the bullet and make another
> attempt at using your persistant storage. 

These are different things.

1. There is an implementation of persistent objects that supports
serialization and deserialization. The object may refer to other objects.
The objects are reference counted and when the last reference disappears
the object, if updated, gets serialized.

This persistence layer can be backed by anything. E.g. a database or by a
direct file I/O with some infrastructure sitting on top of it.

2. There is an implementation of a persistent memory pool where you can
allocate and free memory chunks. It supports stream access if you need to
store something larger than single direct I/O element (block). #1 can sit
on top of #2.

3. There is virtual direct I/O layer that supports transactions and makes
it safe against crashes. #2 can sit on top of #3.

4. There is an implementation of various B-trees (maps) including such that
are similar to a relational table with many keys and many values. #4 can
sit on top of #3.

I cannot tell what of this could be useful for you, because I don't know
what are you trying to do.

> Should I stick to normal records rather than use variants for the persistant storage?

See above.

You can store them directly into the persistent memory pool allocating
storage there. You will need some kind of serialization and desrialization
in the form of stream I/O. The persistent memory pool has stream interface.
If you think that serialization is wasting resources, consider portability.
The same storage file should work on a big-endian and little-endian
machine, right? That's is why you should never use Direct I/O with binary
data or stream attributes. The library provides subroutines to serialize
numeric types. Use them to keep stream I/O portable.

You can use the persistent layer if things are complicated. E.g. objects
dependent on each other. Scopes blurred etc. It is possible to have
persistent objects in situ, e.g. a large container of things, graph, tree,
image. Typically very large objects may be too expensive to deserialize.
They may sit in the persistent storage permanently and be accessed via a
proxy object which is serialized and deserialized instead.

Yes, it is complicated, but nobody said it would not be.

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Re: Truncating Direct IO files

[Randy Brukardt]

"tonyg" <tonythegair@gmail.com> wrote in message 
news:8323246f-eb13-49e8-9e0c-5c5ec1002060@googlegroups.com...
>
>
> I am recording records to disk. Sometimes I replace thes records with a 
> smaller set of records.
>    I could delete the file each time but would prefer to truncate the 
> file. I can't see a way of doing this, can you?

Ada doesn't provide that, mainly because not all popular operating systems 
provide a truncation mechanism for files, and we don't want implementations 
to have to fall back on the very expensive copy the whole file and replace 
mechanism (obviously, if that is acceptable to you, you can write that 
yourself).

It appears that Ada 95 stream files provide truncation on Close and Reset 
(and thus some implementations do implement that, but all surveyed 
implementations did it wrong), but many implementations don't truncate and 
Ada 2005 and later removed the wording that appeared to require it. So one 
cannot depend on that.

                                           Randy.