[lug] Max file size

[D. Stimits]

I can't answer what most of the individual filesystems can or cannot do,
except ext2 CAN do larger than 2 GB, but all applications involved must
be compiled for use with large files. This means anything reading the
filesystem, as well as any shell used to run whatever reads or writes.
To quote another email that was with respect to large file limits on
tcsh shell (which is coded to allow large files, but fails to use the
proper linker options):
<quote author="Nathan Scott from SGI">
Use these: "-D_GNU_SOURCE -D_FILE_OFFSET_BITS=64".  It is the 2nd
part that gives you O_LARGEFILE on open, but GNU_SOURCE is a handy
catch-all for several other useful things.  Refer to the comment
near the top of /usr/include/features.h which describes each flag.

Here's a quick-fire way to test this...
# strace /usr/bin/tcsh -c 'echo > /tmp/blop' |& grep 'open("/tmp/blop"'
open("/tmp/blop", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3

And compiled with the above flags...
# strace ./tcsh -c 'echo > /tmp/blop' |& grep 'open("/tmp/blop"'
open("/tmp/blop", O_WRONLY|O_CREAT|O_TRUNC|O_LARGEFILE, 0666) = 3
</quote>

The strace log will confirm the open has or has not O_LARGEFILE.

D. Stimits, stimits at idcomm.com

John Karns wrote:
> 
> I would like to use dd to create a compressed image file of a vfat
> partition which exceeds 2GB in size, and I bump into the limit which
> aborts the dd process.  What I'm trying to do is write to a reiserfs
> exported via nfs on another box where there is sufficient space.  Ideally
> I would like to fragment the file to burn it to several CD's.  For a
> minute, it looked like the dump utility was the answer, but it will only
> read ext2 partitions.  I'm running SuSE 7.1 with a 2.2.20 kernel.  I have
> already done a back up to tape, but the Adaptec pcmcia SCSI card I have
> maxes out at 1MB/s, making it a bit painful.  I would also like something
> a little more permanent and reliable than tape.
> 
> Seems like I remember reading sometime during the past year or so the max
> file size limit in the kernel had increased substantially.  I waded
> through some of the kernel source docs and headers and saw some reference
> to a maximum size of something like 4TB.  A quick search of Deja brought
> up the following:
> 
> --------------------------
> 
> The max file size is determined by the offset
> used in fseek. Even though fseek has an offset of
> a 64-bit number, the offset is only 32-bits for
> compatibility purposes with older processors.
> Also the offset is signed because an offset can
> be used to go backwards from the current position:
> 
>         fseek(fd, -32L, SEEK_CUR);
> 
> ...
> 
> You can change this to 63-bits by either using fopen64
> and it's corresponding 64-bit functions or by
> compiling source code with the -D_FILE_OFFSET_BITS=64
> 
> Also, the file system itself has limits on the maximum
> file size. Minix is limited to 64 Megabytes while Ext2
> is limited to 2 Gigabytes and xfs has a theoreticat
> limit of 8388608TB.
> 
> --------------------------
> 
> Can anybody fill me in?  Do I need to make the target fs something like
> ext3 in order to accomodate a file size greater than 2GB, or will ext2 /
> reiserfs suffice?  Will simply recompiling dd with the
> -D_FILE_OFFSET_BITS=64 parameter mentioned above do the trick?
> 
> Thanx.
> 
> ----------------------------------------------------------------
> John Karns                                        jkarns at csd.net
> 
> _______________________________________________
> Web Page:  http://lug.boulder.co.us
> Mailing List: http://lists.lug.boulder.co.us/mailman/listinfo/lug