I'm currently needing to ship VM files around between systems, and as you know, VMs that do work-work tend to be a little bit bigger than VMs you've just stood up.
One of my limiting factors is the speed of the network. My shiny new workstation only has a 1Gbit NIC, because I done goofed. But it does have a 9950X.
Raw data off the source:
/source$ time cp source.qcow2 /dest/vmdisk.qcow2 real 30m44.526s user 0m0.337s sys 0m54.131s /dest$ ls -lah -rwxr-xr-x 1 root root 200G May 20 06:59 vmdisk.qcow2 # time tar -czf source.tar.gz vmdisk.qcow2 real 51m31.022s user 49m24.397s sys 1m58.603s
That's a 200Gb disk image, and 51 minutes to compress, even after it's on a Gev4 NVMe drive.
ye olde "tar -czvf" was maxing one core. but I have 16, so why not use 16?
Some numbers
After some googling, I found "--use-compress-program=zstdmt" could speed up the compression some.
I did benchmark around for a little while, but the default settings seemed to be the best, even if all of the options where orders of magnitude faster.
$ time tar -cf vmdiskmt.tar.gz vmdisk.qcow2 --use-compress-program=zstdmt real 3m21.987s user 4m19.242s sys 2m12.028s $ time tar -cf vmdiskmt-T8-M8192.tar.gz vmdisk.qcow2 --use-compress-program="zstdmt -T8 -M8192" real 5m23.804s user 4m18.913s sys 2m18.323s $ time tar -cf vmdiskmt-T16-M16384.tar.gz vmdisk.qcow2 --use-compress-program="zstdmt -T16 -M16384" real 4m22.331s user 4m23.365s sys 2m20.564s $ time tar -cf vmdiskmt-T4-M16384.tar.gz vmdisk.qcow2 --use-compress-program="zstdmt -T4 -M16384" real 4m59.288s user 3m59.898s sys 2m20.753s
In conclusion, drop the "z" from "tar -zcvf" and add a "--use-compress-program=zstdmt" to the end, and you'll save a LOT of time!
