http://varnish.projects.linpro.no/wiki/ArchitectNotes

This is an impressive read........they guy seems to do some smart
things by letting the os kernel do a lot of the work its designed for
and he does some tricks to avoid systems calls and keepign things in
filesystems that are expensive.

Can forth use this level of detail when it exists on linux? How does
forth interact with virtual memory?

Do clever forthers use this kind of optimization?

Good stuff.

According to gavino gmail.com>:

Another way to see this text is the following: this guy read the man
page for the mmap() system call and thought he was Enlightened. He's not
the first. Using a file-backed memory area and relying on the kernel and
the MMU to move things around between the RAM and the disk is not a new
idea. I even did it myself ten years ago when writing a C preprocessor
(I had weird hobbies at that time), and that was no invention of mine
either.

On Apr 28, 1:21 pm, Thomas Pornin bolet.org> wrote:

On Apr 28, 1:21 pm, Thomas Pornin bolet.org> wrote:

Thomas Pornin bolet.org> writes:

Writing a file with mmap() is more complex than reading a file with
mmap(); IIRC you have to specify the file length in advance with
truncate() or ftruncate(), so in some sense you do pre-allocate the
complete file (although on many OSs (f)truncate() alone creates a
sparse file, which does not need much space).

Also, reading a file with mmap() can save a lot of overhead compared
to read() (one copying of the data) or, worse, fread() (two copyings
of the data). But the mmap() advantage is not so pronounced when
writing, because in that case the OS typically writes to disk anyway,
and the copying is cheap compared to that.

According to Anton Ertl mips.complang.tuwien.ac.at>:

That's the problem, actually. On most Unix-like systems, ftruncate()
creates a sparse file, which means that the space is not really
allocated. When the application writes to the pages, the OS will
allocate space for those pages, but if the write are performed somewhat
randomly in the file, then this will lead to severe fragmentation on the
filesystem (filesystem allocation heuristics are tuned for low
fragmentation when new files are written linearly).

You avoid that effect by first filling the file with random data,
linearly, which forces the OS to allocate blocks in a proper way (i.e.
most sequential on the disk). But this means that the complete space is
used, mostly for junk data.

Anton Ertl mips.complang.tuwien.ac.at> wrote:

I am, but I don't think that these really are myths. There's no doubt
in my mind that run-time optimizations can be a spectacular win. It's
hard to compare, because programming styles are so different with Java
and languages that don't usually use GC and aren't type-safe, for
example C++.

Andrew.

Andrew Haley writes:

Thomas Pornin bolet.org> writes:

Why should the application do that? And why should it then write
linearly if it does not use mmap(). If the data arrives non-linearly,
I would expect the application to seek in the file and write randomly
even with the write() interface, resulting in just the same
fragmentation (if any).

According to Anton Ertl mips.complang.tuwien.ac.at>:

File systems with no support for sparse files (e.g. FAT) simply write
zeroes on the "holes" and have no real problem here.

For file systems with support for sparse files, consequences ranges from
relatively benign (data chunks will be spread accross the disk, and you
may have bad interactions with the read-ahead strategies of the kernel
and the logic implemented by the disk itself, which both optimize the
usual linear access pattern) to rather inconvenient (e.g. much more
extra space used to describe the more complex file structure).

Moreover, if your system supports sparse files and has a kernel and a
disk which perform read ahead, then the OS is probably a kind of Windows
or Unix-like system and most plausibly there are parts of the system
which will read the file linearly (if only the backup system) and suffer
from the fragmentation.