Subject:

using fallocate to punch holes in the /nix/store

Message-Id: https://www.5snb.club/w/fallocate-hole-punch/
Linked-From: wiki.

This was an experiment, and it turned out to not be too relevant.

But the script is below

import os

BLOCK_SIZE = 4096
EXPECTED = b"\x00"*BLOCK_SIZE

def scan(f):
    ct = 0
    while True:
        buf = f.read(BLOCK_SIZE)
        if buf == EXPECTED:
            ct += BLOCK_SIZE
        if buf == b"":
            return ct

sum = 0
for (dirpath, _, filenames) in os.walk("/nix/store"):
    for name in filenames:
        fname = os.path.join(dirpath, name)
        if os.path.islink(fname):
            continue
        try:
            with open(fname, "rb") as f:
                s = scan(f)
                sum += s
                if s > 0:
                    print(s, fname)
        except PermissionError:
            pass
        except FileNotFoundError:
            pass
    pass

print(sum)

recommend running it as
python run.py | tee log.txt
so you can inspect what files have aligned runs of zeros.

and if you want to see the total so far,
cat log.txt | jq -R . | jq --slurp . | jq 'map(. | split(" ")[0] | tonumber) | add'

(but note that if the program finished, it prints the total itself as the last line)

On my system, this was around 3.3GB or so. This does double-count if you have hardlink optimization enabled, so I’d say the real savings would be a bit under half that.

so is it worth doing? 🤷.

for reference: the hardlink optimization itself saves 33GB for me.