[lug] Convertion of ext4 Sparse Files

[Lee Woodworth]

The discussion so far appears to use terminology in different ways so I can't tell if:

1) There is a file P that has the bytes for a gpt partition
    formatted with an ext4 file system:
       that contains a file F that is a regular file from the file system point of view
       but has a compressed format that so happens to have the word sparse in its
       name/description (and may also be using techniques similar to what a file system
       does for sparse files)

or

2) There is a file P that has the bytes for a gpt partition
    formatted with an ext4 file system:
       that contains a file S that is a _sparse_ file from the file system point of view
       but curiously also has a file signature that the file command recognizes as an
       "android sparse file"

Case 2 looks weird. If S is a real sparse file then a compressed format on top doesn't
look to be very helpful and may even reduce the number of holes that can be used for
sparse allocation.

How is the inner file used on the target? With just open/read/write via some
clib/java-lib? Through some application-provided VFS layer (e.g. a FUSE mount
in user space)? ...

Precisely how is a "non-sparse" file on a non-target system easier to use?
Loop mount not needed? Standard tools work on the converted file but not the original? ...

On 01/08/2018 12:00 PM, stimits at comcast.net wrote:
> The tool which converted the ext4 loopback-mountable file system seems to be somewhat custom...none of the Linux tools seem to work with this (though "file" recognizes it as an Android sparse file...dump is one of the tools not working with this variation of sparse file). Even Android tools fail. I have a query in with the author of the tool to find out what might be non-standard on how this tool creates sparse from non-sparse.
>   
> ----- Original Message -----From: Bear Giles <bgiles at coyotesong.com>To: Boulder (Colorado) Linux Users Group -- General Mailing List <lug at lug.boulder.co.us>Sent: Mon, 08 Jan 2018 15:39:45 -0000 (UTC)Subject: Re: [lug] Convertion of ext4 Sparse Files
> 
> 
> I can't help with android but for Linux you can use the traditional 'dump' and 'restore'. I wrote a java reader for the dump file (and contributed it to Apache commons) and had to deal with sparse files since those programs are smart enough to recognize them. (I also got to look at the extended attributes, e.g., things like SELinux labels. The apps are smarter than you would expect for something that's essentially unmaintained now.)
> 
> I would suggest looking at any extX packages you can find. The standard tools will always provide data when reading a sparse block. IIRC that includes 'dd' unless you're operating on raw devices. It definitely happens if you mmap a file into your program space. To see the sparse gaps you need to open the raw devices and use the filesystem library.
> 
> 
> On Sat, Jan 6, 2018 at 2:01 PM, <stimits at comcast.net> wrote:
> 
> FYI, the file systems were originally created on ordinary Ubuntu and Fedora systems via a loopback mounted file. This file is copied bit-for-bit exactly to the device...or read back bit-for-bit exact during cloning. It is the conversion from non-sparse to sparse which has the unknown Android issue, but when not sparse it is entirely plain vanilla desktop PC Linux.
>   
> ----- Original Message -----From: Aaron D. Johnson <adj at fnord.greeley.co.us>To: Boulder (Colorado) Linux Users Group -- General Mailing List <lug at lug.boulder.co.us>Sent: Sat, 06 Jan 2018 20:33:47 -0000 (UTC)Subject: Re: [lug] Convertion of ext4 Sparse Files
> 
> 
> 
> stimits at comcast.net writes:> > Search on Android sparse image. I found all kinds of tools> > referenced online.> > In order for reading a sparse file to result in the bytes of a> non-sparse file the reader or the driver of the file system must> recognize this as such...apparently this version is recognized as> an Android sparse image when the "file" command is run on it, but no> other tool...even some of those specific to Android ...seems to work> on this. I'm trying to get more information from the creator of the> tool which created the sparse file.
> 
> Your "Android sparse image" file is something different than a sparsefile on a UNIXy filesystem like ext4. This is unfortunately veryconfusing for those of us trying to help out -- we saw "ext4 sparsefile" in your subject line and went off trying to understand how thatmight cause the problems you're experiencing.
> 
> At this point, you are correct -- the tool which created the Androidimage file (sparse or otherwise) has left you with something thatneeds to be extracted if you'd like to access using a more "normal"(by which I mean desktop or server-ish) Linux system. I would hopethe image creator tool's author is capable of explaining how toextract the image file's contents.
> 
> Please let us know if you can get anywhere on that front. If not,perhaps you can share the image file itself somewhere and others cansee what it might take to make use of it.
> 
> - Aaron_______________________________________________Web Page: http://lug.boulder.co.usMailing List: http://lists.lug.boulder.co.us/mailman/listinfo/lugJoin us on IRC: irc.hackingsociety.org port=6667 channel=#hackingsociety
> 
> 
> _______________________________________________
> 
> Web Page:  http://lug.boulder.co.us
> 
> Mailing List: http://lists.lug.boulder.co.us/mailman/listinfo/lug
> 
> Join us on IRC: irc.hackingsociety.org port=6667 channel=#hackingsociety
> 
> 
> 
> 
> 
> 
> _______________________________________________
> Web Page:  http://lug.boulder.co.us
> Mailing List: http://lists.lug.boulder.co.us/mailman/listinfo/lug
> Join us on IRC: irc.hackingsociety.org port=6667 channel=#hackingsociety
>