Re: compression tricks for storing terabytes of images?

Posted by Roger Leigh on
URL: http://confocal-microscopy-list.275.s1.nabble.com/compression-tricks-for-storing-terabytes-of-images-tp7584921p7584928.html

*****
To join, leave or search the confocal microscopy listserv, go to:
http://lists.umn.edu/cgi-bin/wa?A0=confocalmicroscopy
Post images on http://www.imgur.com and include the link in your posting.
*****

On 21/03/2016 10:37, Emmanuel Levy wrote:
> I'm wondering if some of you may have suggestions regarding archiving large
> image datasets?
>
> So far I've come up to the conclusion that bzip2 is the most convenient
> compression solution to use because it's lossless and it works on any type
> of data (e.g., tif, stacks) and maintains the metadata.
>
> If anyone knows of good and easy to implement alternatives I'll be very
> happy to hear about it.

For archival you might find "xz" results in smaller filesizes
(https://en.wikipedia.org/wiki/Xz).  It's also a lossless compression
algorithm
(https://en.wikipedia.org/wiki/Lempel%E2%80%93Ziv%E2%80%93Markov_chain_algorithm).
  I have used xz compression to archive all my own data (as tar.xz)
without encountering any problems.

Some discussion and benchmarks you might find interesting:

http://unix.stackexchange.com/questions/108100/why-are-tar-archive-formats-switching-to-xz-compression-to-replace-bzip2-and-wha
   https://www.rootusers.com/gzip-vs-bzip2-vs-xz-performance-comparison/
   http://tukaani.org/lzma/benchmarks.html

xz has essentially replaced bzip2 for most general uses, the exception
being plain text (and hence genomic data) for which its compression
algorithm is optimised.  For images, you'll most likely get better
compression and faster decompression with xz; the tradeoff being that it
is a bit slower at compression, but if the goal is the smallest possible
filesize then it's likely worth the tradeoff.

It would probably be worth testing bzip2 and xz at different compression
levels to see how much benefit you see with your own data.


Kind regards,
Roger

--
Dr Roger Leigh -- Open Microscopy Environment
Wellcome Trust Centre for Gene Regulation and Expression,
School of Life Sciences, University of Dundee, Dow Street,
Dundee DD1 5EH Scotland UK   Tel: [phone]

The University of Dundee is a registered Scottish Charity, No: SC015096