[lug] Convertion of ext4 Sparse Files

[stimits at comcast.net]

The file is apparently something non-standard, I'm trying to find out more about the tool which converted it to sparse.
 
Originally there is an empty file full of NULL bytes created via truncate (between 15 and 29GB in size rounded to a 4096 byte block size). That file is covered by loopback. The loopback device has "mkfs.ext4" run on it (usually the host is Ubuntu, but not always...for me it is Fedora). An Ubuntu root partition is copied onto the loopback image. A special utility is run on this loopback mountable file to create a sparse version of this file which is then flashed to embedded hardware through a custom driver (the driver is unaware of data content, it simply streams to eMMC the same as dd...the embedded device has the software which converts from sparse back to non-sparse).
 
The "file" command says the flashed file is an Android sparse file. The utility is named as converting regular files to sparse files ("mksparse"). There is no reverse-from-sparse-to-regular of this available on command line...I had mistaken this sparse file as following the same standards which various other tools use in either the regular Linux world or Android. Apparently the "file" command sees this sparse file as Android sparse...no other tool known sees this as a version of sparse which can be converted to non-sparse. Information from the original sparse tool is going to be required as to how it differs from regular Linux and Android tools.
 
FYI, the embedded system is flashed with the sparse version and somewhere within this hardware (during the flash) the hardware converts sparse back to non-sparse...the content occupies the original space as a GPT partition formatted with ext4 and populated with Ubuntu. One can clone this via dd on the embedded system, or via a flash tool from the original PC host on Linux. Both versions of clone are byte-for-byte exact matches...the clone is a byte-for-byte exact match of the original non-sparse file the sparse tool used for its input. It is possible to guarantee the original non-sparse image and the embedded system correctly convert from the tool's idea of sparse to non-sparse.
 
Unfortunately some people archive and save the sparse version of this for later reference and find they want to loopback mount it without destroying the current system content in order to see this data (I throw out the sparse image and store a "bzip2 -9" compressed non-sparse version...this is quite slow though). A typical non-sparse file is between 15 and 30GB. A typical sparse version is around 1.5GB...it's much easier to store and does not require any extra steps. If this sparse file could be converted back to its full non-sparse image it would become useful for examining without destroying an existing system.
 
----- Original Message -----From: Lee Woodworth <blug-mail at duboulder.com>To: lug at lug.boulder.co.usSent: Tue, 09 Jan 2018 05:58:35 -0000 (UTC)Subject: Re: [lug] Convertion of ext4 Sparse Files

The discussion so far appears to use terminology in different ways so I can't tell if:

1) There is a file P that has the bytes for a gpt partition formatted with an ext4 file system: that contains a file F that is a regular file from the file system point of view but has a compressed format that so happens to have the word sparse in its name/description (and may also be using techniques similar to what a file system does for sparse files)

or

2) There is a file P that has the bytes for a gpt partition formatted with an ext4 file system: that contains a file S that is a _sparse_ file from the file system point of view but curiously also has a file signature that the file command recognizes as an "android sparse file"

Case 2 looks weird. If S is a real sparse file then a compressed format on top doesn'tlook to be very helpful and may even reduce the number of holes that can be used forsparse allocation.

How is the inner file used on the target? With just open/read/write via someclib/java-lib? Through some application-provided VFS layer (e.g. a FUSE mountin user space)? ...

Precisely how is a "non-sparse" file on a non-target system easier to use?Loop mount not needed? Standard tools work on the converted file but not the original? ...
...
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lug.boulder.co.us/pipermail/lug/attachments/20180109/1ffb143b/attachment.html>