Parallel Compression

Data Management
1

What package/command is the CARC recommendation for parallel compression. We used to use pbzip2 but this package doesn’t appear to be installed anymore.

Thanks for any answers,

Luigi–

2

Some options include:

  • pigz (parallel gzip compression, with -p option)
  • xz (with --threads option)
  • zstd (with --threads option)

I think xz will probably give the maximum compression but may take longer than the other options. And zstd has become popular recently because it’s a newer design focused on speed.

3

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.