» 2011 » June

Article posted on Jun 30

New compression for Finnix 102

Posted by Ryan Finnie on June 30, 2011, 2:27 pm

In my previous post, I mentioned switching to xz compression without much explanation.

Software tends to get larger over time. Finnix deals with this in two ways: looking for things to cut while still retaining as much functionality as possible, and just accepting that the size will continue to grow over time. Larger kernels, larger software, etc. The smallest release of Finnix was Finnix 86.2, at 92 MiB (x86). The last release, Finnix 101, was the largest so far at 128 MiB (x86).

During development of Finnix 102, the simple act of adding upgraded kernels and upgrading existing Debian packages to the latest versions was seriously adding to the size of Finnix, nearly 150 MiB at one point. And I had done all I could to reduce the size; there was just nothing left to cut out. Then I found out about xz compression. Xz is based on LZMA2, offers 10-20% more compression than Gzip, and in particular has optimizations for architecture-specific bytecode (program binaries, etc). Xz support was added to Linux 2.6.39, and could be used for kernel, initrd and SquashFS compression (previously, all three used Gzip compression).

So I tried it out, using Linux 3.0-rc (as of right now, if 3.0 comes out soon, it will be included with Finnix 102). I used xz compression for the kernel, initrd and SquashFS root filesystem, and was seeing 20% space savings versus using Gzip.

No compression:
  Root FS:    385,187,840
  initrd:       7,534,080
  Final ISO:  400,162,816

Gzip compression:
  Root FS:    136,921,088 (64.45%)
  initrd:       2,495,967 (66.87%)
  Final ISO:  146,857,984 (63.30%)

Xz (LZMA2) compression:
  Root FS:    109,469,696 (71.58% none, 20.05% gzip)
  initrd:       1,733,032 (77.00% none, 30.57% gzip)
  Final ISO:  118,644,736 (70.35% none, 19.21% gzip)

Overall, it looks great. Though one consideration is performance. LZMA is known for being very processor intensive. LZMA2 is more processor efficient than LZMA (and BZ2), but still slower than Gzip. Compression times are roughly 5 times slower than with Gzip. But the real question is how well decompression would work in the real world (that is, when booting Finnix). I don't have any hard numbers on that, but I did put dev builds through the gauntlet of everything from an i586 AMD K6 to a Power Mac G3 to a Core 2 Quad Q9450, and could not perceive any slowdowns.

So barring any major problems, Finnix 102 will actually be smaller than its predecessor, for the first time in over 3 years.

As a reminder, the standing goal of Finnix is to keep the distribution CD below the size of a Mini CD (185 MiB). While I doubt many people actually use Mini CDs, it's a good round goal, and while Finnix would still be within the 185 MiB limit without xz compression today, a proactive approach to keeping the distribution size low helps it from becoming a crisis too quickly.

By the way, the build scripts have been updated to allow a builder to choose which method of compression (xz, Gzip, or even none), and scripts within Finnix that rely on reading/modifying the initrd (finnix-netboot-server, for example) have been updated to detect which compression method is being used, and will use that method when rebuilding the initrd.

2 Comments
Posted in Development, Finnix

Article posted on Jun 28

Finnix 102 development status

Posted by Ryan Finnie on June 28, 2011, 11:59 pm

Finnix 102 is coming along well. At this point, everything on my list is done, and I'm just waiting for Linux 3.0 to be finalized (I've been testing against 3.0-rc builds for awhile, with good success). A changelog so far (subject to change before final release):

* dist-upgrade
* Upgraded kernels to 3.0 (Debian 3.0.0-xxx)
* Added xen-blkfront to the initrd (AMD64 kernel)
* Added initrd loading attempt of xen-blkfront and xen-netfront
* Added ability to set manual IPv4 addresses and basic resolv.conf info via boot parameters
* Changed kernel, SquashFS compressed root, and initrd to xz compression
* Changed finnix-netboot-server and finnix-netboot-biginit to detect gzip/xz/uncompressed initrd, and build initrd_net accordingly
* Fixed console boot flag
* Fixed automatic update of /etc/fstab during hotplug device update
* Upgraded Memtest86+ to 4.20

The current x86 dev build is 2520, which gave me the idea to go back and look at what builds corresponded to what releases.

* 86.1: 1435
* 86.2: 1504 (+69)
* 87.0: 1617 (+113)
* 88.0: 1776 (+159)
* 89.0: 1961 (+185)
* 89.1: 1969 (+8)
* 89.2: 2011 (+42)
* 90.0: 2084 (+73)
* 91.0: 2154 (+70)
* 91.1: 2185 (+31)
* 92.0: 2270 (+85)
* 92.1: 2296 (+26)
* 93.0: 2318 (+22)
* 100: 2359 (+41)
* 101: 2473 (+114)

Prior to the 86.0 release, I picked build #500 as a rough guess as to how many builds I had made since then. 86.0 itself didn't have the build number shown publicly, so I don't have that data. Most testing is done on a VM (formerly VMware, today VirtualBox), so comparatively few builds are actually burned to CD (I would estimate about 200, which is still a lot). But nearly every build has at least been booted.

PowerPC builds use a different build number which is much smaller, 226 as of today. This is because the primary dev machine is an older G4 Mac Mini, which takes much longer to build compressed roots (formerly about 5 minutes, now over 20 minutes due to the switch to xz compression as mentioned in the changelog), compared to a minute or so on x86, a Core 2 Duo E7200. So most development and architecture-agnostic testing is done on x86 in a VM, then the changes are transferred over in bulk to the PPC environment and built.

Leave Comment
Posted in Development, Finnix

Article posted on Jun 27

Finnix in social networks

Posted by Ryan Finnie on June 27, 2011, 5:46 pm

Finnix now has a page on Facebook, for all your Finnix-related facing and/or booking needs! Please visit facebook.com/FinnixCD today.

There is also a Finnix group on Identi.ca, simply join the group and post with the !Finnix grouptag.

As a reminder, the best place for technical discussions (problems, suggestions, etc) is the Finnix mailing list. Be sure to check the latest release notes first for errata and known problems.