[lug] shmem/mmap/HUGETLB question

[Anthony Foiani]

Doug --

"Doug Pintar" <ratnip3 at gmail.com> writes:

> Copying long files (like movies and stuff, at least several hundred
> MB) from one drive to another takes longer than I think it should.

I see that Jon has already replied, and I agree with most of what he
had to say.

If you still want to beat your head against this, a few leads you
might want to try before attempting to subvert the VM system:

* O_DIRECT.  Oracle was funding work on this for a long time; given
  their interest in getting to the raw bits with as little
  interference as possible, you might want to see if it helps you as
  well.

* As Jon mentioned, "splice()" exists to do pretty much exactly what
  you're trying to do.  Search terms like "zerocopy" and "sendfile"
  might also be helpful.  To whet your appetite:

    SPLICE_F_MOVE: if the output is a file, this flag will cause the
    kernel to attempt to move pages directly from the input pipe
    buffer into the output address space, avoiding a copy operation.

    -- http://lwn.net/Articles/178199/ (Written by Jon, of course. :)

You'll also want to profile (either old-school profiling, oprofile, or
perf) to make sure that you're optimizing the right bits of your
solution.

You might also want to look at "vmstat" and "iotop" while the copy is
executing.

Along the same lines, make sure you get baseline performance
measurements from the two devices you're trying to transfer between.
"hdparm", "bonnie" (and "bonnie++"), and "time dd ..." are all useful
here.  

My understanding of current (late 2011) technology is that single
rotating disks can't generally saturate the device connection bus
(typically SATA at 3 or 6 Gbps), nor the backplane (typically PCI-E x4
lanes @ 2Gbps per lane so 8Gbps).  Some SSD and RAID devices can
saturate SATA links.

In my own experience, doing a simple command-line "cp" of large files
between two single-spindle devices can easily saturate the underlying
hardware (30-40MBps with little CPU load).

A useful suggestion is to place your large file on one RAM-based disk
(e.g., tmpfs) and copy it to another; doing RAM-to-RAM copy should
remove physical device issues.

Anyway.  Looking forward to seeing your results!

Best regards,
Tony