[lug] Convertion of ext4 Sparse Files

Wed Jan 10 01:16:08 MST 2018

I suppose you have already looked at this:
     https://github.com/anestisb/android-simg2img
and it is one of the tools that doesn't work.

On 01/09/2018 05:56 PM, stimits at comcast.net wrote:
> It isn't my image...I was asked how to recover this. One person inherited an embedded system along with some images. The process of cloning and restoring takes many hours to accomplish and requires a specific set of software on a Linux development machine tailored for that. Neither he nor I created the sparse images...but it would be useful if I could find a way for him to turn that sparse image back into a full sized image without spending an entire day backing up the original, then another day per image (remember, this is slow embedded hardware...it isn't a desktop system...an ability to go from sparse to full image on a PC would be a huge time saver).
>   
> Personally I don't have much money (I don't have even a spare $100 at the moment). I have a single SATA port left, and my cable for the power from the modular supply is missing...I already have six drives and all of my fans on a single bus...I wouldn't mind another drive, but this would defeat the purpose of speed since it would still take an entire day...and it wouldn't help the other guy since he is somewhere else in the world and I can't transfer 30GB any faster than he could clone and restore. He doesn't know a lot about Linux, I thought I'd look into it for him.
>   
> ----- Original Message -----From: Lee Woodworth <blug-mail at duboulder.com>To: lug at lug.boulder.co.usSent: Tue, 09 Jan 2018 23:52:03 -0000 (UTC)Subject: Re: [lug] Convertion of ext4 Sparse Files
> 
> So the issue is really about saving disk space. A 3TB 7200rpm 6gbs sata disk is ~$85 atnewegg right now. 2TB USB3 drives are ~$100. If adding local disks would work, people could keep both versions. An inexpensive file server with a compressed file system might also work (e.g. an old dev system).
> 
> If you really need to conserve space then a compressing file system might be of use. BTRFS https://btrfs.wiki.kernel.org/index.php/Compression ZFS https://wiki.archlinux.org/index.php/ZFSThere is also squash fs, but it looks to work best in read-only situations.
> 
> For cases where you want to increase manual file compression speed, pbzip2 or pxz mightbe helpful. These run the compression using multiple threads so it can speed thingsup on multicore machines.
> 
> Just a note, in a linux context, a sparse file is commonly associated with afile-system feature where ls says a file is 30GB, but the file system uses less spaceto store the actual data, e.g. 2GB. This is what I thought of when you mentioned sparsefiles. What you have is _not_ a traditional sparse file, but a compressed regular fileusing a special compression format.
> 
> On 01/09/2018 09:32 AM, stimits at comcast.net wrote:> The file is apparently something non-standard, I'm trying to find out more about the tool which converted it to sparse.> > Originally there is an empty file full of NULL bytes created via truncate (between 15 and 29GB in size rounded to a 4096 byte block size). That file is covered by loopback. The loopback device has "mkfs.ext4" run on it (usually the host is Ubuntu, but not always...for me it is Fedora). An Ubuntu root partition is copied onto the loopback image. A special utility is run on this loopback mountable file to create a sparse version of this file which is then flashed to embedded hardware through a custom driver (the driver is unaware of data content, it simply streams to eMMC the same as dd...the embedded device has the software which converts from sparse back to non-sparse).> > The "file" command says the flashed file is an Android sparse file. The utility is named as converting regular files to sparse files ("mksparse"). There is no reverse-from-sparse-to-regular of this available on command line...I had mistaken this sparse file as following the same standards which various other tools use in either the regular Linux world or Android. Apparently the "file" command sees this sparse file as Android sparse...no other tool known sees this as a version of sparse which can be converted to non-sparse. Information from the original sparse tool is going to be required as to how it differs from regular Linux and Android tools.> > FYI, the embedded system is flashed with the sparse version and somewhere within this hardware (during the flash) the hardware converts sparse back to non-sparse...the content occupies the original space as a GPT partition formatted with ext4 and populated with Ubuntu. One can clone this via dd on the embedded system, or
> 
> via a flash tool from the original PC host on Linux. Both versions of clone are byte-for-byte exact matches...the clone is a byte-for-byte exact match of the original non-sparse file the sparse tool used for its input. It is possible to guarantee the original non-sparse image and the embedded system correctly convert from the tool's idea of sparse to non-sparse.> > Unfortunately some people archive and save the sparse version of this for later reference and find they want to loopback mount it without destroying the current system content in order to see this data (I throw out the sparse image and store a "bzip2 -9" compressed non-sparse version...this is quite slow though). A typical non-sparse file is between 15 and 30GB. A typical sparse version is around 1.5GB...it's much easier to store and does not require any extra steps. If this sparse file could be converted back to its full non-sparse image it would become useful for examining without destroying an existing system.> > ----- Original Message -----From: Lee Woodworth <blug-mail at duboulder.com>To: lug at lug.boulder.co.usSent: Tue, 09 Jan 2018 05:58:35 -0000 (UTC)Subject: Re: [lug] Convertion of ext4 Sparse Files> > The discussion so far appears to use terminology in different ways so I can't tell if:> > 1) There is a file P that has the bytes for a gpt partition formatted with an ext4 file system: that contains a file F that is a regular file from the file system point of view but has a compressed format that so happens to have the word sparse in its name/description (and may also be using techniques similar to what a file system does for sparse files)> > or> > 2) There is a file P that has the bytes for a gpt partition formatted with an ext4 file system: that contains a file S that is a _sparse_ file from the file system point of view but curiously also has a file signature that the file command recognizes as an "android sparse file"> > Case 2 looks weird. If S is a real sparse file then a compressed format on top doesn'tlook to be very helpful and may even reduce the number of holes that can be used forsparse allocation.> > How is the inner file used on the target? With just open/read/write via someclib/java-lib? Through some application-provided VFS layer (e.g. a FUSE mountin user space)? ...> > Precisely how is a "non-sparse" file on a non-target system easier to use?Loop mount not needed? Standard tools work on the converted file but not the original? ...> ...> > > > _______________________________________________> Web Page: http://lug.boulder.co.us> Mailing List: http://lists.lug.boulder.co.us/mailman/listinfo/lug> Join us on IRC: irc.hackingsociety.org port=6667 channel=#hackingsociety>
> 
> _______________________________________________Web Page: http://lug.boulder.co.usMailing List: http://lists.lug.boulder.co.us/mailman/listinfo/lugJoin us on IRC: irc.hackingsociety.org port=6667 channel=#hackingsociety
> 
> 
> 
> _______________________________________________
> Web Page:  http://lug.boulder.co.us
> Mailing List: http://lists.lug.boulder.co.us/mailman/listinfo/lug
> Join us on IRC: irc.hackingsociety.org port=6667 channel=#hackingsociety
>