Re: A Customer's Request For Open Source Software

["Warren W. Gay VE3WWG" <ve3wwg@cogeco.ca>]

I had written a detailed reply to this by my crappy
Netscape 7 had an error posting via NNTP, and threw
the whole reply away (major grumbling here about that!)
So this will be a very brief repeat of the essential
items ;-)

olehjalmar kristensen - Sun Microsystems - Trondheim Norway wrote:
>>>>>>"WWGV" == Warren W Gay VE3WWG <ve3wwg@cogeco.ca> writes:
>     WWGV> You might wonder why buffered "block" devices are not good
>     WWGV> enough for the purpose. I can't answer to the specifics, but
>     WWGV> only that database engines are in a position to better manage
>     WWGV> the cache based upon what they "know" needs to be done. Another
>     WWGV> important reason to control caching details is that when a
>     WWGV> transaction is committed, you need to guarantee that the
>     WWGV> data is written to the disk media (or can be recovered if
>     WWGV> the database is to be restarted at that point).
> 
> The cache management will typically be the same with both OS files and
> raw devices. The main difference is as you say, that you have better
> control over the raw device with respect to the layout of your tables,
> so you may be able to optimize disk writes better.

Control is where it is at. With block devices you can only:

1) leave it up to the O/S when to flush out writes
2) call sync(2) and have all dirty blocks written out
3) call fsync(3) and have all of your own blocks related
    to the file descriptor written out.

With a block device, a return from write(2) does not guarantee
that your data has been recorded on the magnetic media.

With a raw device, it should be a guarantee, with the following
exceptions:

1) Smart caching IDE like disk drives
2) EMC like subsystems with electronic cache


> Most operating systems will allow you to wait until the data are
> even if you are using the file system, so there is no difference with
> respect to the durability of data.

With UNIX, a return from write(2) is only a promise that
it will someday be written. No specific timeline is
guaranteed (see above). I don't expect it is much different
with win32 systems.

> All portable DBMS need to do their own cache management, so if you are
> running on files, both the OS and the DBMS cache the same blocks,
> thereby wasiting RAM. Also, the replacement strategies may be
> conflicting, resulting in suboptimal performance.

Yes, if the DBMS is expecting a raw device, and doing its own
caching, then using a block device (or file system file) is
asking for double-buffering. Very bad indeed for performance.

>     WWGV> The database
>     WWGV> must be recoverable at any given point anyhow, and this
>     WWGV> usually requires fine grained control over physical writes
>     WWGV> to the media. This aspect and performance means that the
>     WWGV> engine must balance performance with reliability (persistance),
>     WWGV> which are conflicting goals when using disk.
> 
>     WWGV> It is this last area where oodles of persistent fast memory
>     WWGV> (instead of disk), can make a world of difference. In this
>     WWGV> case persistence = memory performance, which of course is
>     WWGV> where the win is. If disks became obsolete (one can hope),
>     WWGV> then I could see that new database engine design (internals)
>     WWGV> will become much different than what it is today.  Certainly,
>     WWGV> many of the present compromises would be eliminated.
> 
>     WWGV> -- 
>     WWGV> Warren W. Gay VE3WWG
>     WWGV> http://home.cogeco.ca/~ve3wwg
> 
> Possibly, but keep in mind that most current DBMS's are already CPU
> bound when it comes to throughput. Fast disks and techniques like
> group commit ensures that the log is rarely a bottleneck, and all you
> need to recover is in the log. 

This is primarily where I disagree: I don't disagree that there
are fast disks and subsystems today. But keep in mind two things:

1) There are oodles of traditional slow disk based databases around
    still
2) The database engines were designed to handle the difficult case
    of #1.

Designing a new RDBMS would be much simpler, if you could ignore
the performance cost of putting data to the magnetic media. But
I would venture that all of the popular database vendor offerings
available today, were designed to get performance out of slow
disk I/O technologies.

Otherwise, they could just insist on
this new technology, and make life easier forthemselves from
a design and enhancement point of view.

> What you can get with large non-volatile memory is much lower latency
> per transaction, that is, the response time for a single transaction
> can be dramatically lower, even if the throughput in terms of TPS
> stays the same.

It also buys an opportunity to re-design the database engine,
because now you can eliminate:

  1) the cost of slow disk I/O
  2) the worries about data persistance (no disk
     write operations need to be issued)

This opens the doors to a whole new set of database engine
design dynamics.

> And in case you were wondering if you could do away with the log, the
> answer is yes, if you create some kind of multi-version system. But
> you always need some way of keeping track of the history, so you can
> roll back your changes in case a transaction decides to abort for some
> reason. Actually, one may say that the log IS the database, all the
> rest is there only to give faster access to the latest version.

Going to fast, persistent memory, opens a lot of doors indeed.
-- 
Warren W. Gay VE3WWG
http://home.cogeco.ca/~ve3wwg